4D  A  096 8  7 


RESEARCH  NOTE  80-19 


EXPERIMENTAL  EVALUATION  OF  CONCEPTS  FOR  MX QS TORE: 
AN  ONLINE  INTERACTIVE  LANGUAGE  FOR 
'  TACTICAL  INTELLIGENCE  PROCESSING 


Robert  V.  Katter  and  Gregg  Bell 
Operating  Systems,  Inc. 


HUMAN  FACTORS  TECHNICAL  AREA 


U.  S.  Army 

Research  Institute  for  the  Behavioral  and  Social  Sciences 


December  1979 

Approved  for  public  relent;  distribution  unlimited. 


81  3  26  034 


U.  S.  ARMY  RESEARCH  INSTITUTE 

FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 


A  Field  Operating  Agency  under  the  Jurisdiction  of  the 
Deputy  Chief  of  Staff  for  Personnel 


JOSEPH  ZEIDNER 
Technical  Director 


FRANKLIN  A.  HART 
Colonel,  US  Army 
Commander 


NOTICES 


DISTRIBUTION  Pfinwy  Attribution  ot  tbit  report  bat  baan  madt  by  ARI.  PKatt  addre u  corratpondtnct 
eon  earning  dittnbution  ol  report!  to  U.  S.  Army  Aataarcb  Inttitutt  tor  tba  Babavioral  and  Social  Setanaat, 
ATTN:  PER  IT? ,  5001  Eitanbotmr  Avanua,  Alexandria,  Virginia  32333. 


FINAL  DISfOtlTIOft  Thl*  "eort  may  ba  dattroyad  urban  it  it  no  lengar  itaadad.  Pieett  do  not  return  it  to 
tba  U.  S.  Army  Retaareb  inttitutt  tor  tba  Babavioral  end  Social  Sciences. 


NQTJI  Th«  I'ndmgt  m  tb.t  report  are  not  to  be  conn  rued  at  an  otticiel  Debertment  or  tba  Army  potmen, 
uniatt  to  datignatad  by  otbar  autbornad  documtntt 


maintaining  a  useful  level  of  ueer/operator  fm 
essential  elements  of  the  Interactive  language 
for  the  display  arrangement.,. 


illlarlty  with  little-used  hut 
The  results  were  promising 


3  & 


Unclassified 

tccueiTv  Cb*smrie*riOM  or  this  r  Aoar*t*«n  omm  am«« 


EXPERIMENTAL  EVALUATION  OF 
CONCEPTS  FOR  MIQSTURE;  AN 
ONLINE  INTERACTIVE  LANGUAGE 
FOR  TACTICAL  INTELLIGENCE 
PROCESSING 


r 

Prepared  for: 

U.  S.  Army  Research  Institute 
for  the  Behaviqral  and  Social 
Sciences 
(PERI -OS) 

5001  Elsenhower  Avenue 
Alexandria,  VA  22333 


OS I:  R79-050 
10  December  1979 


Operating  Systems,  Inc. 


21031  Vtntun  Boulevard,  Woodland  HUh,  CA  91364 


1 


CONTENTS 


1.  FORWARD. 


1-1 


2.  OVERVIEW. 


2-1 


2.1  Background - - - — • — • —  2-1 


3.  TASK  KNOWLEDGE  FOR  TASK-STEP  DISPLAYS . 


3-1 


3.1  Introduction......... 


3-1 


3.2  Background ........ 


3-1 


3.3  Method  Summary 


3-2 


3.4  Detailed  Procedures  and  Materials. 


3-3 


3.5  Presentation  of  Results. 


3-4 


3.6  Discussion  of  Implications., 


3-28 


4.  EVALUATION  OF  A  FAMILIARIZATION/REFRESHER  DISPLAY  IN 


MIQSTURE _ 4-1 


4.1  Introduction. 


4-1 


4.2  Problem  Background _ _ 


.■•••••••••••••••••••••••••■•••••••••••••••••••••••••••••••••••••••••••a* 


4-1 


4.3  Problem  Statement..—. ............. 


>ee#eeeeeeeeeeeee eeeeeeeeeeeeeeeeeeeeeeeeeeee eeeeeeeeeeeeeeees 


4-2 


4.4  A  Research  Approach , 


4-3 


4.6  Results. 


• •• • eee ee •• ee ease aeeee eeeee • eeeee e e • • • 


4-6 


-  I  - 


/ 


4.6  Discus* ion 


5.  Conclusion  ... 

6.  APPENDICES 


EXPERIMENTAL  EVALUATION  OF  CONCEPTS  FOR  MIQSTURE;  AN  ONLINE 
INTERACTIVE  LANGUAGE  FOR  TACTICAL  INTELLIGENCE  PROCESSING 

Robert  V.  K  attar 
Greg  Bell 

Operating  Systems,  Inc. 


ABSTRACT 


A  project  was  undertaken  to  evaluate  selected  aspects  of  an  online 
language  for  Army  tactical  intelligence  processing  called  MIQSTURE.  In  the 
first  of  two  experiments  reported  here,  U.  S.  Army  tactical  intelligence  off¬ 
icers  provided  expert  judgments  on  work-related  and  information  utiliza¬ 
tion  aspects  of  descriptions  of  selected  tasks  from  Army  tactical  intelli¬ 
gence  processing.  The  results  provided  indications  of  what  query  methods 
have  potential  as  aids  for  intelligence  analysts. 

In  the  second  experiment,  an  evaluation  was  made  of  the  efficacy  of  a 
famillarizaticn/refresher  display  arrangement  for  developing  and  maintain¬ 
ing  a  useful  level  of  user/operator  familiarity  with  liftle-used  but  essential 
elements  of  the  interactive  language.  The  results  were  promising  for  the 
display  arrangement. 

1.  FORWARD 


This  report  covers  the  second  year  of  work  on  a  project  to  investigate  the  potentials 
of  the  mixed  initiative  type  of  interactive  language  for  Army  tactical  intelligence  pro¬ 
cessing  applications.  During  the  first  year,  a  conceptual  model  for  the  language  was 

developed  and  evaluated  by  expert  judgment  methods.  In  the  second  year,  selected 

/ 

aspects  of  the  language  were  studied  in  the  field  and  laboratory.  In  a  mixed  initia¬ 
tive  interaction  language,  both  the  user/operator  and  the  system  contribute  active 
initiatives  to  maintaining  a  successful  dialog  to  support  ongoing  interactive  tasks. 
The  potential  advantages  of  the  mixed  initiative  type  of  interactive  language  are 
especially  important  for  Army  tactical  intelligence  processing.  Foremost  is  the  fact 
that  interaction  with  battlefield  automated  systems  (of  which  tactical  intelligence 
systems  are  one  example)  requires  the  processing  of  highly  diverse  information  in  a 
variety  of  contexts.  Under  such  circumstances,  system  users  have  a  great  deal  to 
remember.  This  Is  especially  true  as  the  pace  of  modern  warfare  radically  Increases 
the  quantities  of  enemy  and  friendly  information  that  must  be  handled  while  at  the 
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•an*  ttm*  curtailing  the  time  available  for  procasaing  the  information  for  timely 


decision-making. 


2.  OVERVIEW 


This  report  presents  prslimlnsry  rsssarch  to  evaluate  aspects  of  an  online  interac¬ 
tion  language  for  user/operator-system  Interfaces  for  Army  tactical  intelligence  pro¬ 
cessing  systems  called  MIQSTURE  (for  Mixed  Initiative  Query  Structure  with  Task  and 
liaer  Related  Elements).  The  first  study  addressed  the  problem  of  securing  design 
Information  from  potential  usars  for  features  of  the  language  that  can  help  cue  the 
user/operator  through  complex  task  sequences.  The  second  study  investigated  the 
effectiveness  of  learning  arrangements  In  CRT  displays  for  helping  user/opefators  to 
learn  necessary  alternative  entry  modes  for  boolean  expressions  and  record  element 
qualifiers  that  are  used  to  search  a  file  of  records. 

/ 

2.1  Background 

The  MIQSTURE  concept  anticipates  the  development  of  tactical  intelligence  systems 
using  large  automated  data  bases  with  data  from  a  variety  of  sources.  With  such  a 
system,  the  Intelligence  analyst  could  roam  purposefully  through  data  and  ask  a 
series  of  questions:  "Where  are  enemy  anti-aircraft  radars  located  now?"  “What 
were  their  positions  twelve  hours  ago?"  "How  many  main  enemy  supply  routes  have 
been  Identified?"  "From  what  coordinates  have  there  been  reports  of  enemy  armor 
activity?"  Using  queries  such  as  these,  the  analyst  could  develop  an  understanding 
of  the  current  situation  and  use  It  to  produce  an  intelligence  product.  A  mixed- 
initiative  system  would  provide  active  support  for  the  analyst’s  task.  However,  such 
support  depends  on  having  a  computer  system  which  has  "knowledge"  of  task  vari¬ 
ables  and  relevant  data. 

The  mixed  Initiative  type  of  Interactive  language  emphasizes  computer-inltisted  con¬ 
tributions  to  the  human-computer  dialog.  Both  the  machine  subsystem  and  the 
user/operator  play  active  roles  in  contributing  initiatives  to  the  interactive  dialog. 
The  computer  actively  contributes  by,  for  example,  volunteering  information  appropri- 
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ate  to  the  infant  of  a  query;  by  “stepping*  the  uaar/oparator  through  a  complex 
series  of  task  steps  by  prompting  and  providing  cues  and  directions;  by  diagnosing 
user/operator  errors  through  sensing  tha  interactive  context;  by  providing  highly 
specific  "context  sensitive"  feedback  for  errors  and  for  user/operator  requests  for 
help;  by  monitoring  tasking  input  streams  and  notifying  the  user/operator  of  items 
relevant  to  his/her  assignments;  by  embedding  associative  learning  display  mechan¬ 
isms  in  message  presentation  formats  and  command  term  abbreviation  arrangements; 
by  volunteering  online  tutorials  as  part  of  help  procedures;  by  keeping  a  record  of 
ongoing  transactions  with  easy  immediate  re-dispiay  for  the  user/operator;  by  allow¬ 
ing  flexible  multiple  means  for  inserting  parameters  for  most  commands;  by  allowing 
various  forms  of  individualization  of  the  language  for  eachoiser;  and,  by  automatically 
invoking  the  various  Individualized  features  when  a  user/operator  signs  on  the  sys¬ 
tem. 

From  tha  above  examples  it  can  be  seen  that  the  two  main  characteristics  of  a 
mixed-initiative  language  are: 

1 .  To  provide  support  to  the  user’s  limited  memory  for  the  terms  and  conventions  of 
the  interactive  language.  These  include  ability  to  accept  commands  and  aarame- 
ter  specifications  In  memory-supportive  formats  such  as  menu  selection  and  fill- 
in-the-blank  formats,  and  term-tolerance  arrangements  for  the  use  of  synonyms, 
idiomatic  forms  of  expression,  and  correction  of  typographical  and  spelling  errors. 
Menu  selection  and  filMn-the-biank  formats  can  sharply  reduce  the  requirements 
for  recalling  the  exact  "correct"  forms  of  Input  expressions,  requiring  instead 
that  the  user  only  recognize  and  select  the  desired  form  when  it  is  displayed. 
Term-tolerance  arrangements  allow  for  alternative  keyed-in  strings  of  charac¬ 
ters  (synonyms  or  idiomatic  expressions)  to  be  "recognized  "  as  serving  the 
same  communicative  function  by  the  system,  thus  reducing  the  load  on  the  user’s 
memory  of  recalling  only  one  precise  form.  The  same  type  of  arrangements  can 
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be  used  to  have  tno  system  make  approximative  “guesses"  it  me  correct  term 
intended  for  a  misspelled  term  or  a  typographical  error.  When  a  successful  guess 
is  made  by  the  system  (which  is  usually  the  case)  the  user  need  only  indicate 
this  by  a  single  keystroke,  rather  than  re-keying  the  entire  misspelled  term. 

2.  To  anticipate  user/operators’  needs  for  other  types  of  memory  aids  related  to 
the  structure  and  sequence  requirements  of  complex  tasks  performed  interac¬ 
tively.  In  the  latter  roie  the  system  is  provided  with  a  range  of  corrective  infor¬ 
mation  that  it  can  make  available  in  the  event  the  user/operator  falls  to 
remember  a  complex  sequence  of  steps  in  a  task.  Such  aids  are  provided  by  the 
system  on  three  bases: 

—  The  user/operator  may  recognize  a  specific  need  and  provide  the  system  a 
"name"  or  description  of  the  type  of  aid  needed 

—  The  user/operator  may  signal  that  a  need  for  memory  support  exists,  but  be 
unable  to  specify  the  required  form  of  help—  in  which  case  the  system 
attempts  to  diagnose  the  probable  range  of  needs  on  the  basis  of  sensing 
the  current  state  of  the  interactive  context,  and  offers  a  selection  to  choose 
from; 

—  Without  being  asked,  the  system  may  sense  a  probable  need  for  memory  sup¬ 
port  to  the  user/operator  on  the  basis  of  the  status  of  the  current  interactive 
context,  and  offer  unobtrusive  aid  which  the  user/operator  may  accept  or 
ignore. 

The  research  reported  here  was  designed  to  determine  whether  selected  features 
of  the  MIQSTURE  language  are  compatible  with  tactical  intelligence  processing.  The 
first  step  was  to  collect  information  regarding  user  (e.g.,  analyst,  commander) 
requirements.  Then,  features  of  MIQSTURE  specifically  designed  for  the  intelligence 
process  were  explored  and  evaluated.  The  two  types  of  features  selected  for 
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study  in  the  prsssnt  project  were:  display*  for  prompting  th#  ussr  through  complsx 
sequences  of  staps  in  an  intsractiva  taak,  and  displays  for  learning  nacassary  alter¬ 
native  forms  for  entering  queries  used  to  search  files  of  record*. 
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3.  TASK  KNOWLEDGE  FOR  TASK-STEP  DISPLAYS 

3.1  Introduction 

The  task-step  prompting  display  features  of  MIQSTURE  were  proposed  as  a  way  to 
make  the  interactive  computer  program  more  of  a  partner  with  the  user  in  accom¬ 
plishing  complex  tasks  (MIQSTURE:  AN  EXPERIMENTAL  ONLINE  LANGUAGE  FOR  ARMY 
TACTICAL  INTELLIGENCE  INFORMATION  PROCESSING.  ARI  Technical  Report  No.  TR- 
78-A25,  July  1978.  (AO  A064  323). 

While  the  human  task  performer  has  In  mind  purposes  and  possible  outcomes  and 
past  history  of  an  ongoing  task  process,  the  program  is  devoid  of  such  knowledge. 

This  major  shortcoming  of  the  machine  complicates  and  limits  the  user/system  dialog. 

/ 

One  objective  of  the  present  research  was  to  develop  and  test  procedures  which 
would  allow  the  machine  to  accept  and  store  task-structure  knowledge,  and  to  use  it 
In  the  course  of  Interactive  dialogs. 

3.2  Background 

Task  knowledge  is  knowledge  about  a  particular  task.  The  task  knowledge  we  wish 
to  provide  our  computer  system  has  several  characteristics  important  from  the  point 
of  view  of  system  design: 

1 .  It  must  be  provided  by  persons  intimately  familiar  with  the  details  of  each  iden¬ 
tifiable  task. 

2.  It  must  be  constrained  to  reliable  generalizations  about  relations  between  given 
sets  of  conditions  and  the  task  steps  and  substeps  they  imply  or  involve.  Varia¬ 
tions  In  relations  between  conditions  and  task  steps  must  be  expressed  In  une¬ 
quivocal  terms  or  else  cannot  be  part  of  the  machine’s  task  knowledge. 


3.  The  quality,  level  of  detail,  and  usefulness  of  task  knowledge  for  the  system  Is 
dependent  upon  the  effectiveness  of  the  means  provided  to  inform  the  system. 


The  present  pilot  research  was  undertaken  to  provide  an  initial  Indication  of  specific 
information  that  intelligence  analysts  require  to  perform  intelligence  tasks  In  a  non- 
automatad  (manual)  environment.  These  data  were  gathered  to  determine  if  there  is 
a  useful  degree  of  consensus  among  analysts  about  intelligence  processing  subtasks 
and  procedures,  which  could  provide  guidance  for  query  language  deaign.  To  imple¬ 
ment  the  research,  opinions  were  sought  from  analysts  about  selected  procedures 
used  to  process  Intelligence. 

3.3  Method  Summary 

The  requirements  for  the  pilot  research  were  that  the  procedures  must  be  focused 

on  the  intelligence  area  and  be  appropriate  to  the  skill/knowledge  of  available  parti- 

/ 

cl  pants.  To  satisfy  the  requirements,  task  descriptions  to  serve  as  stimuli  were 
eeiected  with  the  advice  of  staff  personnel  at  the  U.  S.  Army  Intelligence  Center  and 
School  (USAICS),  Ft  Huachuca,  Arizona,  and  from  documents,  including  The  Combat 
Electronic  Warfare  intelligence  Operations  Center  Automated  Functional  Analysis 
Study  (CEWIOCAFAS),  and  USAICS’  Hierarchical  plus  Input  Process  Output  Charts. 
Fourteen  officers  from  advanced  courses  In  army  tactical  Intelligence  at  USAICS 
ware  Identified  by  staff  personnel  as  having  experience  appropriate  to  the  selected 
task  descriptions.  Each  officer  in  the  sample  contributed  two  and  one-half  hours  to 
the  research. 

The  officers  participated  in  groups  of  two  (In  one  case  three),  for  each  session.  The 
sessions  were  conducted  In  five  steps: 

1.  Initial  explanations  and  examples  were  provided  to  the  participants. 

2.  Each  participant  then  read  completely  through  the  task  descriptions,  and  ranked 
them  according  to  degree  of  familiarity. 

3.  The  first  task  in  the  booklet  (See  Appendix  A,  task  2. 1.2.1. —Determine  Source 
Reliability)  was  then  used  for  practice,  without  regard  to  familiarity  rankings  for 


that  task.  In  working  tha  example,  participants  filled  out  a  stngla  MIQSTURE 
roaponaa  form  (Appendix  B)  for  that  taak  with  guidanca  from  tha  project  taam  as 
needed. 

4.  Participants  then  proceeded  to  fill  out  one  additional  form  for  each  task  descrip¬ 
tion  they  had  ranked  as  familiar  (2,  3,  or  4,  depending  upon  the  rate  at  which 
they  worked).  The  first  form  waa  for  their  most  familiar  task,  the  second  was  for 
the  next  most  familiar  task,  and  so  on. 

6.  After  completing  the  response  forms,  a  group  debriefing  discussion  was  held 
among  the  two  or  three  participants  and  the  two  or  three  project  team  members. 

3.4  Detailed  Procedures  and  Materials 

/ 

The  Task  Description  Booklet  (Appendix  A)  and  the  MIQSTURE  response  form  (Appen¬ 
dix  B)  were  the  only  materials  used  by  research  participants.  Explanation  of  the 
research  purposes  coincided  with  major  points  made  in  the  introductory  section  of 
this  report.  They  were  given  in  an  Informal  manner,  usually  occupying  about  ten 
minutes  followed  by  questions  and  clarifications.  Ir.  the  last  part  of  the  introductory 
step  participants  were  asked  to  read  the  explanation  on  tha  cover  page  of  the 
response  booklet  (see  Appendix  B).  i.t  the  booklet,  the  Items  taken  from  the 
CEWIOCAFAS  appear  on  the  left  half  of  each  page;  corresponding  items  on  the  right 
side  of  each  page  are  descriptions  of  hypothetical  automated  versions  of  the  tasks 
developed  by  the  project  team.  The  automated  versions  were  included  to  provide 
participants  with  concrete  examples  of  the  ways  In  which  manual  tasks  of  the 
CEWIOCAFAS  might  be  carried  out  using  an  automated  system.  The  purpose  was  one 
of  stimulating  interest  and  new  perspectives  for  the  judgments  called  for  In  the 
MIQSTURE  response  form. 

Responses  were  made  directly  on  the  form  provided,  and  all  items  except  number  ten 
had  a  six-point  rating  scale.  One  task  description,  concerning  intelligence  collection 
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management,  wu  adsdnlstsrsd  only  to  the  single  participant  who  had  exceptional 
faariiiartty  with  that  task.  Data  from  a  fourtaanth  participant  wara  unusable  bacausa 
tha  tasks  that  wara  ratad  wara  not  Msntfftad  on  tho  rasponsa  torn.  Rating  scaia 
descriptions  differed,  depending  on  tha  purpoaa  of  tha  ttaai.  At  tha  bottom  of  ovary 
page,  a  ptaoa  was  provtdad  for  comment*.  For  many  of  tha  (team,  apaoa  alao  wu 
provided  for  addtttonal  clarification  about  a  particular  rasponsa. 

Project  taaai  mamhara  wara  avalabM  for  quaations  during  tha  aasaton.  but  vary  faw 
raquasta  wara  made  for  clarifications.  Raspondanta  wara  fraa  to  taka  a  braak 
whanavar  thoy  daairad.  Whan  all  participants  In  a  sasaion  had  coaiplatad  thatr 
raaponsa  forma,  a  braak  waa  auggastad,  and  than  a  dabrtoflng  dlacuaslon  waa  hald 
lasting  for  twenty  minutes  to  mora  than  an  hour.  Discussion  was  Informal  and  open 
ended,  but  tha  project  team  assured  that  tha  fodoadng  points  wars  oovarad: 

a  Old  tha  participants  have  any  relevant  experiences  or  opinions  to  share  with 
tho  pro/oct  team? 

a  What  wore  tho  participants'  depressions  of  tha  session:  DM  they  find  tha 
teaks  difficult  or  easy;  If  so,  which  parts?  CouM  they  sea  any  ways  In  which 
such  activities  might  Jreneflt  tha  Army;  If  so  how? 

a  DM  tha  participants  have  any  opinions  about  automating  Army  tactical  kiteW- 
gence  processing? 

M  Presentation  of  Results 

Results  are  considered  in  tho  following  order: 

1 .  Task  FamWarfty  Rankings 

2.  Frequency  of  Non-Responses  for  It  mn  Combined  across  Tasks 
S.  Frequency  Distributions  of  ratings  for  Items  by  Individual  Task 
A.  Comments  on  Items  for  Individual  Tasks 


6.  Debriefing  Comments 


3£.  1  Taak  Familiarity  Ranking* 

Data  for  Task  Familiarity  ara  summarized  in  Table  1. 


BKLT 

SEQN 

CEWtOC 

NUMBER  NAME  OF  TASK  DESCRIPTION 

CHOICE  MEAN 
FREQ.  RANK 

3 

2.1 .2.3 

Post  CoNateral  Eneaw  Order  of  Battle 

11 

1.18 

1 

2.1. 2.1 

Determine  Source  Reliability 

6 

2.00 

6 

2. 1.2.5 

Maintain  Frten<*y  Situation  Overlay 

6 

2.00 

4 

2.1. 2.4 

Maintain  InteMpiwic*  Workbook 

4 

3.00 

8 

3. 1.2.1 

Determine  Resource  Availability 

4 

2.76 

2 

2. 1.2.2 

Post  Electronic  Enemy  O.of  B.  Map 

3 

2.00 

0 

3. 1.2.2 

Determine  Resource  Capability 

3 

2.33 

0 

2. 1.2.0 

Maintain  Tampiating  File 

3 

2.00 

10 

3.1. 2.3 

Identify  Other  Sources 

2 

3.00 

12 

3.1. 1.2 

Determine  Indicators 

'  1 

2.00 

7 

3.1. 1.4 

Review  Collection  Planning  FHe 

1 

3.00 

Table  1 :  Task  Familiarity  Ranking* 


Proceeding  from  the  loft,  the  first  three  column  headings  of  Table  1  show  the 
eaqusnoa  of  tasks  In  the  task  description  booklet,  ths  CEWIOCAFAS  classlficstion 
masbers  for  a  task,  and  the  associated  task  name.  Task  descriptions  are  rank- 
ordered  from  the  top  to  the  bottom  of  the  table  by  their  overall  familiarities  to  the 
InteMigencs  officer  participants  in  the  research.  The  Choice  Freq.  column  depicts  the 
number  of  participants  (in  the  total  sample  of  thirteen  officers)  who  ranked  that  task 
description  as  fssMiar  enough  to  be  chosen  among  the  ones  to  be  rated  later  in  the 
session.  The  Mean  Rank  ookimn  shows  the  arithmetic  mean  of  familiarity  rankings  for 
each  task  description,  with  the  value  1.0  signifying  the  greatest  famWartty,  and  0.0 
Mtoatlng  least  famiiarity.  The  number  of  participants  for  Task  2. 1.2.1—  Determine 
Source  Reliability— was  actually  thirteen,  because  that  task  was  used  as  ths  work- 
through  example  by  all  participants.  However,  the  number  of  participants  who  chose 
the  task  on  the  basis  of  their  preceding  task  familiarity  rankings  was  five,  which  Is 
the  choice  frequency  indicated  In  Table  1. 
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The  booklet  sequence  position  of  task  description  3.1.1. 2-- Determine  Indicators--  is 
given  me  12  in  Table  (which  Is  its  position  as  shown  in  Appendix  A).  However,  for 
twelve  of  the  thirteen  participants,  only  the  first  ten  task  descriptions  (through 
3. 1.2.3— Identify  Other  Sources)  were  shown  because  of  previously  obtained  Infor- 
istion  about  the  relative  familiarities  that  participants  had  with  the  various  tasks 
described. 

When  the  three  most  familiar  task  names  (top  three  In  Table  1 ),  are  compared  with 
the  three  least  familiar  ones  (bottom  three),  there  is  an  apparent  difference  in  qual¬ 
ity.  The  more  familiar  tasks  seem  associated  with  more  immediate  processing 
urgency  In  terms  of  placing  incoming  Information  into  an  ovaratl  context  ss  soon  as 
possible,  provided  the  Information  is  reliable.  The  least  "familiar  tasks  appear  to  be 
sssoclated  with  activities  that  may  be  somewhat  lass  time-pressured,  such  ss  plan¬ 
ning  efficient  future  information  gathering,  and  assuring  the  maximum  feasible  cover¬ 
age.  Given  that  this  Interpretation  is  valid,  these  results  indicate  that  the  task  fami- 
larity  rankings  of  the  Intelligence  officer  participants  were  based  on  a  common 
viewpoint  about  which  activities  are  of  highest  priority  in  performing  Intelligence 
analysis  and  which  therefore  receive  more  attention.  Such  results  could  also  have 
been  an  artifact  due  to  the  order  of  presentation  In  the  task  booklet  (with  earlier 
descriptions  being  rated  more  familiar).  The  circumstances,  however,  do  not  lend 
strong  support  to  this  Interpretation,  because  ail  Items  were  read  through  before 
famWartty  was  rated.  In  fact,  booklet  sequence  numbers  are  only  moderately  corre¬ 
lated  with  the  familiarity  rank  order  of  Items  shown  In  the  table  (r».06).  In  summary, 
analyst  orientations  toward  task  value  and  priority  appeared  to  be  reasonably  con¬ 
sistent  between  participants,  and  were  determinable  by  the  kinds  of  techniques 
used  in  this  study. 

3.6.2  Analysis  of  Raaponaa  Form  Items  for  Tasks 


The  columns  In  Table  2  show,  from  laft  to  right,  item  numbers  (corresponding  to  their 
positions  in  the  response  form),  frequencies  of  NO  ANSWER  responses  for  each  item, 
and  an  abbreviated  text  for  each  Item.  Logical  sub-parts  of  a  task  are  referred  to 
as  “steps"  or  "task  steps",  items  appear  in  the  table  in  the  rank  order  of  their  NO 
ANSWER  frequencies. 

Table  2:  Frequency  of  non-responses  to  Items 


ITEM 

NO. 

ummmmmmm 

N/A 

FREQ 

ABBREVIATED  TEXT  ITEM 

■nnnnintnnmiiimiummiinnmnnti 

How  EASY  is  It  to  identify  data 
needed  from  earlier  steps? 

12 

14 

13 

11 

How  READILY  ACCESSIBLE  are  needed 
data  from  earlier  steps? 

10 

0 

Is  this  step  dependant  on  com¬ 
pletion  of  other  steps?  Describe. 

14 

8 

How  USEFUL  Is  task  step  checkoff 
capability  as  a  CURRENT  reminder? 

11 

7 

How  OFTEN  are  earlier  step  data 

NECESSARY  to  the  present  step? 

16 

7 

To  store  task-step  OUTCOMES,  how 
useful  is  menu  type  checkoff? 

16 

6 

Hoto  USEFUL  is  task  step  checkoff 
capability  as  LATER  REVIEW  store? 

a 

6 

Are  activities  in  TASK  STEP  right 
in  relation  to  surrounding  steps? 

18 

6 

How  USEFUL  for  step  outcome  store 

Is  free  typing  entry  capability? 

(continued  next  page) 
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17 

4 

How  USEFUL  for  step  outcome  store 
is  blank  fill-in  capability? 

0 

4 

In  thia  STEP,  the  NUMBER  of  Items 
to  be  RECALLED  by  MEMORY  is: 

6 

4 

Is  this  STEP  concerned  with  NON- 
DEFERRABLE  actions  on  data? 

4 

a 

Can  the  ACTIVITIES  of  thia  STEP 
be  ELIMINATED  if  neceaaary? 

8 

3 

In  this  STEP,  the  NUMBER  of  ttema 
needing  CONCURRENT  ATTENTION  la: 

7 

3 

How  DEMANDING  ia  this  STEP  in 
requiring  your  TIME  and  THOUGHT? 

8 

3 

Can  the  ACTIVITIES  of  this  STEP 
be  SHORTENED  or  DEGRADED  if  need? 

2 

3 

Please  comment  on  DETAIL  OF 

DESCRIPTION  of  task  STEP: 

1 

0 

How  USEFUL  Is  task  STEP  DESCRIPTION 
for  CUEING  and  PROMPTING? 

The  rank-order  clusterings  of  Items  in  Tabia  2  suggest  a  patterning  in  the  responses. 
Starting  at  tha  top  of  die  table  is  a  group  of  items  (12,  13,  10)  which  received 
fewer  responses  than  all  other  Items.  The  three  Items  of  this  cluster  asked  for 
responses  about  relationships  among  task  steps  in  task  processes.  At  the  bottom  of 
the  table  appears  a  cluster  of  the  first  two  (easiest  to  answer)  items  (1,  2),  which 
asked  for  opinions  about  the  descriptive  TEXTS  for  the  tasks.  Tha  succeeding  six 
Items  (e.7,8,4,6,0)  ask  for  evaluative  opinions  about  tasks  themselves.  Finally,  the 
reaieining  sequence  of  Items  (17,18,3,16,10,11,14)  are  about  an  equal  mixture  of 
two  kinds  of  items;  those  asking  for  evaluative  opinions  about  the  usefulness  of 
interactive  capabilities  for  keeping  track  of  tasks,  and  those  asking  for  opinions 
about  the  relattonahipa  among  task  steps  In  more  inclusive  task  processes. 

As  a  working  hypothesis,  there  seem  to  be  a  systematic  pattern  In  the  degrees  of 


3-8 


1 


confidence  with  which  the  intelligence  officer  participants  answered  questions  about 
Intelligence  processing  tasks  and  the  implications  for  automation.  An  alternate  possi¬ 
bility  is  that  results  for  the  rank-order  of  frequencies  were  an  artifact  of  serial  posi¬ 
tions  of  the  items  in  the  response  form.  This  is  unlikely,  since  there  is  only  a  slight 
relationship  (r*.32)  between  the  rank  order  of  NO  ANSWER  response  frequencies  of 
items  and  their  serial  positions  in  the  response  form.  Overall,  the  response  form 
emerges  as  a  reasonable  source  of  data  about  the  intelligence  process. 

3.5.3  Ratings  of  Response  Items  for  Individual  Tasks 

Distributions  of  ratings  on  given  items  were  developed  for  each  task  description. 
Individual  items  with  responses  for  each  of  the  tasks  are  shown  in  Tables  3  through 
18.  Appendix  C  summarizes  the  same  data  separately  for  each  task.  Since  the 
average  response  frequencies  for  individual  tasks  was  small  due  to  the  limited 
number  of  participants,  no  statistics  were  computed  and  results  were  analyzed  using 
visual  comparisons  of  data  distributions.  Ratings  for  various  tasks  on  each  item  are 


discussed  together. 


Task 

NA 

1 

2 

3 

4 

6 

6 

2. 1.2.3 

Post  Collateral  Enemy  0  of  B.  _ 

6 

2 

1 

2 

2.1. 2.1 

Determine  Source  Reliability  ,  , 

1 

1 

1 

T 

2. 1.2.6 

Malnt.  Friendly  Sit.  Overlay 

T 

1 

~T~ 

1 

2.1. 2.4 

Maintain  Intelligence  Wrkbook_  _  . 

1 

T 

'  r — 

3.1. 2.1 

Determine  Resource  Available  _ 

“1 

~T 

T' 

'“i  * 

2.1. 2.2 

Post  Eltmc  0  of  B  Battle  Map 

‘  T" 

r 

T' 

3.1. 2.2 

Determine  Resource  Capability 

-~r~ 

"1 

2.1. 2.6 

Maintain  Templating  Files  _ 

i 

1 

i 

3.12.3 

Identify  Other  Sources 

i 

i 

3.1. 1.2 

Determine  Indicators.  . 

i 

3.1. 1.4 

Review  Collection  Plan  File _ 

i 

Table  3  (Item  1 ) 

Item  1:  "How  usmfut  is  this  task  step  description  for  cueing  and  prompting  the  per¬ 


former  about  what  to  do?"  1  --Extremely  useful,  to  6— Of  little  use. 


All  participants  rated  this  item— perhaps  indicating  considerable  participant  confi¬ 
dence  in  providing  an  answer  to  It.  However,  there  was  only  scattered  consensus 
(Le.,  no  tight  frequency  clusters  within  rating  distributions)  for  the  eleven  tasks. 
Responses  to  task  description  2.1 .2.3— (Post  Collateral  Enemy  Order  of  Battle) 
clustered  around  a  rating  of  "extremely  useful”,  with  a  possible  trend  in  this  direc¬ 
tion  for  task  3.1 .2.1 --(Determine  Resource  Availability).  Most  other  distributions  were 
scattered,  with  the  number  of  responses  too  small  to  discern  any  clear  pattern. 


Task 


NA  1  2  3  4  5  6 


2.1. 2.3 
2.1 .2.1 
2.1 .2.6 

2.1. 2.4 

3.1 .2.1 

2.1. 2.2 

3.1. 2.2 
2.1 .2.6 

3.1. 2.3 
3.1. 1.2 

3.1. 1.4 


Post  Collateral  Enemy  0  of  B _ _ _ 

Determine  Source  Reliability _ 1 _ 

Malrrt  Friendly  Sit.  Overlay _ 

Maintain  Intelligence  Wrkbook _ 

Determine  Resource  Available _ 

Poet  Eltmc  O  of  B  Battle  Map _ 

Determine  Resource  Capability _ 

Maintain  Templating  Files _ 

Identify  Other  Sources _ 

Determine  Indicators _ 

Review  Collection  Plan  File _ _ 


7  4 

I  2 — ~B — 3 - T 

T — I - 2 — T - 

- 7 - T 

1 — 7 — I - 1 - 

- 3 - 

T - 7 - 

1  I 

~ _ 1 _ 


Table  4  (Item  2) 

Hem  2:  "Please  comment  on  the  LEVEL  OF  DESCRIPTION  of  the  task  step."  1  —Too  Gen¬ 


eral,  to  6— Too  Detailed. 


i 
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Most  ratings  for  the  item  clustered  near  the  scale’s  midpoint,  with  some  tendency  in 
the  direction  of  "too  general".  None  of  the  individual  task  rating  distributions  cluster 
close  to  either  extreme  of  the  scale. 


Task 

NA 

i 

2  3 

4 

6 

6 

2.1 .2.3 

Post  Collateral  Enemy  0  of  B 

2 

8 

1 

2.1. 2.1 

Determine  Source  Reliability.  _ 

1 

2 

6 

r 

i 

2.1 .2.5 

Moint.  Friendly  Sit.  Overlay 

3 

i 

1 

2.1. 2.4 

Maintain  Intelligence  Wrkbook.. 

3 

3.1 .2.1 

Determine  Resource  Avsllable 

T 

‘1  ' 

i 

1 

2. 1.2.2 

Post  Eltme  0  of  B  Battle  Map 

2 

'  r~ 

3. 1.2.2 

Determine  Resource  Capability  _ 

'  T 

,_r- 

T  ‘ 

2. 1.2.6 

3.1 .2.3 
3.1. 1.2 

3.1. 1.4 

Maintain  Templating  Files _ 

Identify  Other  Sources _ 

Determine  Indicators _ 

Review  Collection  Plan  File _ 

1 

_l 

— 

l 

/ 

._2_. 

1  ~ 

— 

”  ■ 

— 

Table  5  (Item  3) 

item  3s  "Are  the  task  step  BOUNDARIES  (beginning,  end)  correct  In  relation  to  the 
other  task  steps  surrounding  It?"  1—Many  activities  in  this  step  should  be  put  in 
other  steps,  to  O—Many  activities  in  other  steps  should  be  put  in  this  step. 


The  modal  response  to  this  item  across  all  tasks  falls  at  "Just  about  right" ,  with  the 
occasional  possibility  that  tails  of  Individual  task  distributions  would  favor  one  or 
another  scale  extreme  if  more  cases  were  available. 


Task 

NA  1 

2 

3 

4 

6 

6 

2.1 .2.3 

Post  Collateral  Enemy  0  of  B 

i 

1 

1 

1 

7 

2.1. 2.1 

Determine  Source  Reliability 

2 

7“ 

1 

1 

2. 1.2.5 

Maint.  Friendly  Sit  Overlay 

2 

'1  ' 

2 

2.1. 2.4 

Maintain  Intelligence  Wrkbook_ 

1“ 

r 

3. 1.2.1 

Determine  Resource  Available.. 

T 

r 

z  J  " 

2.1. 2.2 

Post  Eltme  0  of  B  Battle  Map 

“T" 

~2  ' 

3.1 .2.2 

Determine  Resource  Capability  . 

1 

2 

2. 1.2.6 

Maintain  Templating  Files 

1 

2 

3. 1.2.3 

Identify  Other  Sources  .  . 

1 

1 

3.1. 1.2 

Determine  Indicators  _  . 

1 

3.1. 1.4 

Review  Collection  Plan  File _ 

r~ 

Table  6  (Item  4) 
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Item  4:  " Can  the  activities  of  this  stop  bo  eliminated  if  necessary?"  1  —Always,  to  6- 
-  Never. 

Thar*  ar*  some  noticeable  trends  toward  clustering  for  some  tasks.  Examples  are  a 
"never"  duster  for  task  2.1 .2.3— (Post  Collateral  Enemy  Order  of  Battle),  a  “some¬ 
times"  duster  for  2. 1.2.1—  (Determine  Source  Reliability),  and  possible  "never"  clus¬ 
ters  for  most  other  tasks.  The  only  deviation  from  a  "never"  rating  occurred  for  the 
Determine  Source  Reliability  task.  It  Is  explained  by  written  comments  on  the  forms 
and  verbal  comments  in  the  debriefing  sessions!  Participants  suggested  that  the  reli¬ 
ability  of  most  commonly  used  sources  Is  pre-determined  and  quite  constant  over  a 
given  period  of  time,  and/or  Is  determined  by  personnel  nearer  the  data  sources  in 
most  cases.  From  this  point  of  view,  determining  source  reliability  would  often  be 
relatively  superfluous  for  many  known  message  sources,  in  general,  all  of  the  tested 
tasks  seam  to  be  necessary  parts  of  the  intelligence  analyst’s  job. 


Task 

NA 

i 

2 

3 

4 

<0 

to 

2.1. 2.3 

Post  Collateral  Enemy  0  of  B _ 

1 

_ 5__ 

4 

..... 

2. 1.2.1 

Determine  Source  Reliability 

L. 

_  J 

L_ 

1 

Jl 

L  _ 

2.1. 2.6 

Maint.  Friendly  Sit.  Overlay 

_  2__ 

1 

L_ 

l 

2.1. 2.4 

Maintain  Intelligence  Wrkbook_ 

_ JL. 

JL. 

3.1.2.1 

Determine  Resource  Available _ 

1 

2. 1.2.2 

Post  Eltmc  0  of.  B  Battle  Map 

__  I 

3.1. 2.2 

Determine  Resource  Capability _ 

1 

r 

1 

2. 1.2.6 

Maintain  Templating  Files 

1 

1 

T 

3.1. 2.3 

identify  Other  Sources  L  . 

„  1 

l 

3.1.1 .2 

Determine  Indicators 

i 

3.1. 1.4 

Review  Collection  Plan  File _ _ 

_ 1_ 

Table  7  (Item  5) 

Item  5:  "is  th/s  step  concerned  with  non-del  err  able  actions  (he.,  with  processing 
crucial,  perishable  data)?  1— Always,  to  6— Never. 


This  Item  appears  to  distinguish  somewhat  among  tasks.  The  ratings  for  2. 1.2.3— 
(Post  Collateral  Enemy  Order  of  Battle)  appear  to  cluster  well  toward  the  "always" 
portion  of  the  scale,  as  do  the  data  for  2.1 .2.2— (Post  Electronic  Enemy  Order  of  Bat- 


tie),  and  2.1.2.6--(Maintain  Templating  Fites).  However,  ratings  for  2.1.2.1-- 
(Determine  Source  Reliability)  are  bimodal,  having  one  subgroup  of  the  distribution 
well  over  toward  the  "never”  scale  position,  and  another  subgroup  toward  the 
"Always"  rating.  This  latter  result  fits  well  with  the  earlier  item  4  findings  for  that 
task.  The  data  suggest  that  opinions  on  deferring  many  of  the  other  tasks  might  also 
prove  to  be  variable  among  participants,  but  there  are  insufficient  response  frequen¬ 
cies  to  support  any  firm  conclusion. 


Task 

NA  1 

2 

3 

4 

5 

6 

2.1. 2.3 

Post  Collateral  Enemy  0  of  B 

1 

1 

4 

3 

2 

2. 1.2.1 

Determine  Source  Reliability _ 

1 

1 

6 

3 

1 

2.1. 2.5 

Maint.  Friendly  Sit.  Overlay _ _ 

1 

-  2 

i 

1 

2. 1.2.4 

Maintain  Intelligence  Wrkbook _ 

1 

1 

1 

1 

3. 1.2.1 

Determine  Resource  Available _ 

1 

1 

2 

2. 1.2.2 

Post  Eltmc  0  of  B  Battle  Map 

1 

1 

1 

3. 1.2.2 

Determine  Resource  Capability..,. 

1 

2. 1.2.6 

Maintain  Templating  Files. 

l 

r 

1 

3.1. 2.3 

Identify  Other  Sources. 

’1 

i 

3.1. T.2 

Determine  Indicators 

~T~ 

3.1. 1.4 

Review  Collection  Plan  File _ 

_ 

•_T 

Table  8  (Item  6) 

Item  6:  "Can  the  activities  of  this  step  be  shortened  or  degraded  If  necessary ?"  1  — 
Always,  to  6— Never.  / 

For  this  item,  the  rating  distributions  show  considerable  scatter  for  most  tasks,  sug¬ 
gesting  that  there  may  be  somewhat  more  differences  in  points  of  view  about  shor¬ 
tening  task  activities  than  about  deferring  or  eliminating  them.  Note,  for  example, 
that  results  for  Tasks  2. 1.2.3  and  2. 1.2.1  on  this  item  disagree  with  data  from  Items 
4  and  5. 

One  possible  reason  for  this  variability  is  the  unfortunate  use  of  the  word  "degrade" 
in  the  item’s  text.  This  may  have  carried  a  strong  negative  connotation  to  some 
judges  and  not  to  others.  Another  possibility  is  that  some  participants  knew  ways  to 
shorten  work  activities  without  seriously  affecting  results,  while  others  d*d  not. 


Task 

NA 

1 

2 

3 

4 

6  6 

2.1 .2.3 

Post  Collateral  Enemy  0  of  B 

1 

"2 

5 

1 

1 

1 

2. 1.2.1 

Determine  Source  Reliability _ 

‘I 

2 

r 

4  3 

2.1 .2.6 

Maint.  Friendly  Sit.  Overlay  _  _ 

“i 

I” 

3"  -  " 

2. 1.2.4 

Maintain  Intelligence  Wrkbook_  . 

— r 

T 

7 

3. 1.2.1 

Determine  Resource  Available 

■  1 

r 

7 

2.1. 2.2 

Post  Eltmc  0  of  B  Battle  Map _ 

"  T“ 

'  r 

1  ■ 

3. 1.2.2 

Determine  Resource  Capability  _ 

r 

r~ 

"1 

2.1 .2.6 

Maintain  Templating  Flies 

1 

2 

3.1. 2.3 

Identify  Other  Sources 

~1_ 

_ i_ 

3.1. 1.2 

Determine  Indicators  _ 

1__ 

3.1. 1.4 

Review  Collection  Plan  Rle _ _ _ 

1 

Table  9  (Item  7) 

Item  7:  "How  demanding  is  this  task  step  In  requiring  your  time  and  thought?"  1  - 
Extremely  demanding,  to  6 — Extremely  un-demanding. 


This  Item  appeared  to  differentiate  among  tasks.  The  tasks  rated  as  more  demanding 
were:  2.1. 2.3— (Post  Collateral  Enemy  Order  of  Battle);  2.1. 2.4 — (Maintain  Intelli¬ 
gence  Workbook);  2.1 .2.2— (Poat  Electronic  Enemy  Order  of  Battle  Map),  and 
2. 1.2.0— (Maintain  TeaiplatJng  File).  Taaka  rated  as  less  demanding  were:  2.1. 2.1  — 
(Determine  Source  Reliability);  and,  2.1 .2.6— (Maintain  Friendly  Situation  Overlay). 

Some  of  the  task  steps  that  showed  differences  for  rating  Item  4  (Can  the  step  be 
eliminated?)  showed  similar  deferences  on  Item  7  (How  demanding  of  thought  and 
time  Is  the  step?).  Ratings":*  how  that  the  Posting  of  Collateral  Enemy  Order  of  Battle 
should  not  be  eliminated  and  that  it  Is  usually  quite  demanding  on  thought  and  time. 
Similarly,  Maintaining  Intelligence  Workbook  and  Posting  Electronic  Enemy  Order  of 
Battle  Map  cannot  be  eliminated  and  are  very  demanding  on  time  and  thought.  In 
contrast.  Determination  of  Source  Reliability  can  sometimes  be  eliminated  and  is  not 
so  demanding  of  time  and  thought.  The  correlation  between  results  in  Items  4  and  7, 
however,  does  not  necesserlly  indicate  that  more  demanding  tasks  are  also  more 
necessary. 


Two  alternative  Interpretations  of  this  general  pattern  can  be  made.  One  is  that  the 
experience  of  most  of  the  pertlcipants  has  been  mostly  with  tasks  characterized 


/ 
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above  as  "high-priority",  and  that  this  is  mainly  what  determines  their  ratings  of 
those  tasks  as  having  been  moat  demanding  of  their  time  and  thought  The  other  pos¬ 
sibility  is  that  the  participants  have  had  approximately  equal  amounts  of  experience 
with  all  the  tasks  presented,  and  that  rating  the  demandingness  of  tasks  reflects 
their  true  differential  nature.  The  present  data  do  not  allow  an  unequivocal  choice  to 
be  made  between  the  two  Interpretations. 


Task 

NA 

1 

2 

3 

4 

5  6 

2. 1.2.3 

Post  Collateral  Enemy  0  of  B 

1 

3 

3 

2 

1 

1 

2. 1.2.1 

Determine  Source  Reliability 

1 

2 

1 

3 

3  3 

2.1. 2.5 

Maint.  Friendly  Sit.  Overlay 

1 

2 

2 

2.1 .2.4 

Maintain  Intelligence  Wrkbook__ 

_  2 

_L_ 

3. 1.2.1 

Determine  Resource  Available 

1 

L  "t 

_1 

L 

2. 1.2.2 

Post  Eltmc  0  of  B  Battle  Map 

_JL 

3.1.22 

Determine  Resource  Capability 

7~r 

'  1 

1 

2.1. 2.6 

Maintain  Templating  Files _ _ 

3 

3. 1.2.3 

Identify  Other  Sources 

1 

"l 

3.1. 1.2 

Determine  Indicators 

1 

3.1. 1.4 

Review  Collection  Plan  File _ _ 

T 

Table  1 0  (Item  8) 

Item  8:  In  this  step,  the  number  of  different  Items  needing  more  or  less  simultaneous 
attention  Is.-".  1—Very  Large,  to  8— Very  Small. 


Task  steps  2.1. 2.3— (Post  Collateral  Enemy  Order  of  Battle),  and  2. 1.2.4— (Maintain 
Intelligence  Workbook)  show  tendencies  to  cluster  around  "very  targe",  while  task 
steps  2. 1.2.1. —(Determine  Source  Reliability),  2. 1.2.5— (Maintain  Friendly  Situation 
Overlay)  and  2. 1.2.6— (Maintain  Templating  Files)  show  tendencies  to  cluster  around 
medium-small.  Ratings  of  tasks  on  the  number  of  items  requiring  simultaneous  atten¬ 
tion,  on  Item  7,  demands  for  time  and  thought,  and  on  Item  6,  task  non-deferrability, 
all  appear  to  tap  a  somewhat  common  perception  of  the  tasks.  However,  data  varia¬ 
bility  suggests  that  the  three  kinds  of  ratings  are  not  identical. 
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Task 

NA  1 

2 

3 

4 

5 

6 

2.1. 2.3 

Post  Collateral  Enemy  0  of  B_ 

1 

1  _  . 

_A— 

1 

2.1. 2.1 

Determine  Source  Reliability  _ 

.  1 _ 

1  L_ 

.2 _ 

-4_ 

2 

2. 1.2.6 

Malrrt  Friendly  Sit.  Overlay. 

1 

7 

2.1. 2.4 

Maintain  Intelligence  Wrkbook. _ 

Determine  Resource  Available  _ 

1 

_1 _ 

_1_ 

3.1 .2.1 

L.  L_ 

1 

2. 1.2.2 

Post  Eltmc  0  of  B  Battle  Map _ 

1 

3.1. 2.2 

Determine  Resource  Capability  _ 

1 

I" 

2.1. 2.6 

Maintain  Tem plating  Files _ 

l  l 

1 

3.1 .2.3 

Identify  Other  Sources _ 

1 

1 

3.1. 1.2 

Determine  Indicators _  _ 

l 

3.1. 1.4 

Review  Collection  Plan  File _ _ 

1 

Table  1 1  (Item  0) 

Item  9:  * In  this  step,  the  number  of  Items  that  must  be  recalled  BY  MEMORY  isi".  7— 


Very  large,  to  6— ‘Very  small. 


Task  2.1. 2.3— (Post  Collateral  Enemy  Order  of  Battle)  shows  rating  clusters  toward 
"very  large",  while  task  2.1 .2.5— (Maintain  Friendly  Situation  Overlay)  clusters 
toward  "small”.  The  reasons  for  these  differences  were  not  apparent  from  comments. 
Examination  of  distributions  for  other  tasks  suggests  that  ratings  for  this  Item  may 
be  more  scattered  than  for  the  previous  two  Items  (demand  on  time  and  thought; 
steps  needing  simultaneous  attention).  A  possibility  is  that  a  larger  sample  might 
show  that  consensus  on  judgments  about  memory  demands  is  less  than  for  judgments 
about  time,  thought,  and  attention  demands.  Although  It  is  clear  that  memory 
demands  often  may  be  quite  great  In  Intelligence  information  processing  and  analysis, 
It  may  be  that  memory  functioning  (and  malfunctioning)  Is  "automatic"  for  the  most 
part,  and  ordinarily  escapes  the  notice  of  analysts. 
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Task 


NA  1 


8 


2. 1.2.3  Post  Collateral  Enemy  0  of  B 

1 

Y-10 

N-0 

2. 1.2.1  Determine  Source  Reliability 

1 

Y-8 

N-4 

2. 1.2.5  Maint.  Friendly  Sit.  Overlay  _ 

Y -T 

2.1. 2.4  Maintain  Intelligence  Wrkbook__ 

1 

Y-1 

_N«2 

3.1. 2.1  Determine  Resource  Available 

_ l 

.  YiL 

J4?l_ 

2.1 .2.2  Post  Eltrnc  0  of  B  Battle  Map _ 

1 

Y-l  _  _ 

._  N-l  . 

3. 1.2.2  Determine  Resource  Capability  . 

_  L  _ 

Y-2_ 

2. 1.2.6  Maintain  Templating  Files 

1 

Y=2 

N-0 

3. 1.2.3  Identify  Other  Sources 

l 

Y-l 

N-0 

3. 1.1. 2  Determine  Indicators 

Y-0 

N-l 

3.1. 1.4  Review  Collection  Plan  File _ 

Y-l 

N-0 

Table  12  (Item  10) 

Item  10:  "Is  this  step  dependent  on  the  completion  of  other  steps?  Yes/ No.  Describe 


prerequisite  step(s).  Why  Is  the  step  prerequisite?" 


Although  the  response  frequencies  are  very  small,  about  three  times  as  many  of  the 

r 

tasks  appear  to  be  judged  as  dependent  on  completion  of  earlier  steps  as  were 
Judged  to  be  independent  of  earlier  atepa.  Task  2. 1.2.3— (Post  Collateral  Enemy 
Order  of  Battle)  and  task  2.1 .2.1 --(Determine  Source  Reliability)  were  consistently 
judged  as  dependent  on  the  completion  of  ot  jteps.  The  CARRYOVER  of  data  and 
results  from  task  step  to  task  step  in  intelligence  processing  may  create  extra  bur¬ 
dens  on  the  analyst  to  manipulate  materials  and  data.  Such  demands  on  the  short¬ 
term  memory  of  analysts  mipht  be  reduced  by  the  use  of  automation  aids  which  help 
in  sequencing  and  storing  information. 

Some  of  the  main  dimensions  of  task  step  dependencies  are  Illuminated  by  commen¬ 
taries  concerning  prerequisite  steps  and  the  reasons  they  are  prerequisite.  Seven 
commentaries  mentioned  as  prerequisites  the  referencing  and  maintenance  of  Enemy 
Order  of  Battle  information  (file  version,  workbook  version,  journal  version)  for  pur¬ 
poses  of  putting  new  enemy  Information  into  an  enemy  organizational  ID  context. 
Four  commentaries  mentioned  determination  of  data  and/or  source  reliability  as  a 
prerequisite  to  further  use  of  the  data.  Four  mentioned  track  correlation  processing 
of  new  data  as  a  prerequisite  to  prevent  duplication,  redundancy  and  clutter,  and  to 
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•How  removal  of  older  data  points  on  tha  sama  anamy  objacta  and  actlvitiaa.  Two 
com  manta  Has  mentioned  maaaaga  logging  and  control  as  prerequisites,  and  two  men¬ 
tioned  checkoff  for  Essantial  Elamants  of  Information  and  Other  Required  Information. 
From  tha sa  commants,  It  appears  that  different  participants  tend  to  recall  different 
kinds  of  prerequisites.  This  suggests  that  a  checklist  of  prerequisite  types  might 
wail  produce  more  consistent  data  between  participants. 


Task 

NA 

1 

2 

3 

4 

6 

e 

2.1. 2.3 

Post  Collateral  Enemy  0  of  B  _ 

1 

3 

l 

”T 

~2~ 

....... 

2.1 .2.1 

Detormina  Source  Reliability 

‘“3 

— 7~ 

T* 

T 

~r~ 

~T~  • 

2. 1.2.5 

MairrL  Friendly  Sit.  Overlay 

.  L. 

2 

i 

1 

2. 1.2.4 
3.1. 2.1 

Maintain  Intelligence  Wrkbook_ _ 

Determine  Resource  Available 

3~ " 

1 

/ 

_ 

_ 

— 

2.1. 2J2 

Post  Eltmc  O  of  B  Battle  Map 

1 

i 

i 

3. 1-2.2 

Determine  Resource  Capability  _ 

1 

1 

i 

2. 1.2.6 

Maintain  Tamp  la  ting  Files 

1 

r 

3.1 .2.3 

Identify  Other  Sources _ 

'1 

~T~ 

3.1. 1.2 

Determine  Indicators  _ 

~1~ 

3.1. 1.4 

Review  Collection  Plan  File _ _ 

~T~ 

Table  13  (item  11) 

if  am  11:  "How  OFTEN  are  such  earlier-developed  data  NECESSARY  to  tho  present 


step?"  1 --Always,  to  0— Never. 


Scanning  of  the  response  'distributions  reveals  no  tendencies  for  tight  clustering, 
with  the  possible  exception  of  an  "always"  cluster  on  task  step  3.1. 2.1—  (Determine 
Resource  Availability).  For  task  2. 1 .2.3— (Post  Collateral  Enemy  Order  of  Battle) 
responses  between  item  10  and  1 1  are  somewhat  Inconsistent.  For  item  10  no  parti¬ 
cipants  rated  the  task  on  the  "never"  half  of  tho  scale,  but  4  participants  rated  the 
task  on  tha  "never"  half  of  the  scale  for  Item  1 1 ,  which  shows  a  bipolar  distribution 
for  tha  task.  Nona  of  the  commants  appear  to  shed  any  light  on  this  inconsistency. 


i 
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Task  NA  1  2  3  4  6  6 

2. 1.2.3  Poat  Collateral  Enemy  0  of  B _ 5 _ <L _ 2 _ ! _ !. 


2.1 -2.1  Determine  Source  Reliability  _ 

1 

1  JL..2  1  1 

2. 1.2.6  MaJnt  Friendly  Sit  Overlay.  _ 

2  1 

2. 12.4  Maintain  Intelligence  Wrkbook_. 

_  1 

1  1  1 

3. 1-2.1  Determine  Resource  Available 

_  1 

2  1 

2.1. 2.2  Post  Eltmc  0  of  B  Battle  Map 

.  X _ 

.  _  1  1 

3. 1.2-2  Determine  Resource  Capability. 

1  1  .  1  . 

2. 1-2.6  Maintain  Tempiating  Flies 

1 

1  1 

3. 1.2.3  Identify  Other  Sources 

1 

"  “  i  —  '  '  ' 

3.1. 1.2  Determine  Indicators 

i 

3.1. 1.4  Review  Collection  Plan  File 

l . . '  ' 

Table  14  (Item  12  ) 

Item  12:  "How  EASY  Is  It  to  Identify  the  kinds  of  data  needed  from  earlier  steps?" 

7  — Very  easy,  to  6— Very  difficult. 

Scanning  of  the  response  distributions 

ShOWS 

few  extreme  clusters.  There  were 

"Vary  easy”  clusters  for  task  steps 

3. 1.2.1 

—(Determine  Resource  Availability), 

2. 1.2.3 — (Poat  Collateral  Enemy  Order 

of  Battle),  and  2. 1.2.1 —(Determine  Source 

Reliability),  and  a  "Very  difficult"  rating  from  the  single  participant  who  rated  task 

step  3.1.1 .2— (Determine  Indicators).  These  results,  though  based  on  minimum  data. 

appear  to  make  sense  gh/en  the  demands  of  these  task-steps  compared  to  others  in 

the  study. 

Task 

NA 

1  2  3  4  5  6 

2. 1.2.3  Post  Collateral  Enemy  0  of  B 

3 

3  12  11 

2.1 -2.1  Determine  Source  Reliability,..! 

3 

T  T~  T  "5  T 

2. 1.2.5  Maint.  Friendly  Sit.  Overlay..  _ 

“I - 

1  2  1 

2. 1.2.4  Maintain  Intelilgence  Wrkbook_ 

1  T  '  I  1 - -  ' . . 

3. 1.2.1  Determine  Resource  Available 

"I 

'  - - '2"  1  “ 

2.1. 2.2  Post  Eltmc  0  of  B  Battle  Map  _ 

1  1 

3. 1.2.2  Determine  Resource  Capability.. 

*1  . 1'  "  1  . '  ' 

2. 1.2.6  Maintain  Tempiating  Files_ 

" .  1  I 

3. 1.2.3  Identify  Other  Sources 

— i — 

1 

3. 1.1. 2  Determine  Indicators  _ 

1 

3. 1.1. 4  Review  Collection  Plan  File _ 

T 

Table  15  (Item  13) 

Item  73;  "Ho w  READILY  ACCESSIBLE  are  the  needed  data  from  earlier  steps  (e.g..  In 
Bitmaps,  message  files,  your  memory,  etc.)?”  7  —Readily  accessible,  to  6— 
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InaocmstJble. 


Scanning  of  the  raaponsa  distributions  suggests  that  thla  Item  was  batter  than  Item 
12  at  discriminating  between  task  steps.  The  response  distributions  for  task  steps 
2. 1.2.3 --Post  Collateral  Enemy  Order  of  Battle)  and  2.1. 2.6— (Maintain  Friendly  Situa¬ 
tion  Overlay)  show  clustering  trends  In  the  region  of  "usually  accessible",  while  tha 
dietrtbution  for  step  3. 1.2.1—  (Determine  Resource  Availability)  suggests  a  trend 
toward  "aoaiawhat  Inaccessible".  The  pattern  for  most  other  tasks  with  three  or 
store  data  points  reflects  considerable  scattering  of  opinion. 


Task 

NA 

i 

2 

3 

4 

6 

6 

2.1. 2.3 

Post  Collateral  Enemy  0  of  B_  _ 

1 

i 

1 

'  4 

2 

2 

2.1. 2.1 

Determine  Source  Reliability 

2 

_r~ 

“T" 

1 

T“ 

T“ 

~T~ 

2.1 .2.6 

Maint.  Friendly  Sit.  Overlay 

1 

1 

1 

1 

1 

2.1 .2.4 

Maintain  Intelligence  Wrkbook 

T 

1 

2 

3.1. 2.1 

Determine  Resource  Available 

"1 

~r 

1 

~r 

2.1 .2.2 

Post  Eltmc  0  of  B  Battle  Map 

1 

r~ 

1 

3.1 .2.2 

Determine  Resource  Capability 

“T" 

~I 

I 

2.1 .2.6 

Maintain  Temolatina  Files 

r 

T 

1 

3.1 .2.3 

Identify  Other  Sources 

n 

T 

3.1. 1.2 

Determine  Indicators 

~T' 

3.1.1 .4 

Review  Collection  Plan  File _ 

~T~ 

Table  16  (Item  14) 

Item  74;  "For  the  task  perfdrmer,  how  useful  would  It  be  to  check  off  each  step  In  a 


task  as  an  aid  for  remembering  that  the  step  was  accomplished,  deferred,  deleted, 
short-cut,  ate?"  1— Highly  useful,  to  6— Of  little  use. 


Task  2.1 .2.1— (Determine  Source  Reliability)  shows  a  clustering  tendency  toward 
"very  useful",  while  task  2.1. 2.2— (Post  Electronic  Enemy  Order  of  Battle  Map)  sug¬ 
gests  a  cluster  around  "of  little  use".  Other  tasks  show  no  discernible  clusters. 
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Task 


NA  1  2  3  4  5  6 


2.1. 2.1 

Determine  Source  Reliability _ 

L 

.j _ L_ 

_1  _ 

2. 1.2.5 

Maint.  Friendly  Sit.  Overlay 

,_i _ 1  . 

L 

2. 1.2.4 

Maintain  Intelligence  Wrkbook 

1 

1 

2 

1 

3.1.2.1 

Determine  Resource  Aveileble _ 

1 

1 

1 

2.1. 2.2 

Post  Eltmc  0  of  B  Bettis  Map 

1 

1 

1 

3.1^2 

Determine  Resource  Capability 

"i  r 

T  ' 

2- 1.2. 6 

Maintain  Temolating  Files., 

F 

1 

■ 

“T 

3.1. 2.3 

Identify  Other  Sources 

'1  ' 

3.1. 1.2 

Determine  Indicators,  _ 

T  ' 

3.1. 1.4 

Review  Collection  Plan  File 

I 

Table  17  (Item  15) 

Item  ISi  "How  useful  would  a  check  off  system  (as  In  Item  14)  be  for  LATER  review 
and  critique  of  ones  OWN  performance?"  Responses  were  about  equally  split 

between  "highly  useful"  and  "of  little  use".  Distributions  for  tasks  2.1 .2.4— (Maintain 

/ 

Intelligence  Wbrkbook)  and  3. 1.2.1— (Determine  Resource  Availability)  suggest  clus¬ 
ters  toward  "of  little  use". 


Task 

NA 

1 

2 

3 

4 

5 

6 

2.1 .2.3 

Post  Collateral  Enemy  0  of  B 

2 

2 

2 

3 

1 

1 

2.1. 2.1 

Determine  Source  Reliability 

1 

"  7~ 

■~T~ 

7" 

"TT 

2 

2.1. 2.6 

Maint.  Friendly  Sit.  Overlay 

-  L_ 

3 

2.1. 2.4 

Maintain  Intelligence  Wrkbook_ 

1 

1 

1 

'  T~ 

3.1 .2.1 

Determine  Resource  Available  _ 

'  1 

1 

~T 

1 

2.1. 2.2 

Post  Eltmc  O  of  B  Battle  Map 

1 

3.1. 2.2 

Determine  Resource  Capability 

~  r 

'  r* 

"1 

2.1. 2.6 

Maintain  Templatlrjg  Files 

i 

'1  ' 

"i "  *- 

3. 1.2.3 

Identify  Other  Sources 

_ L_ 

_1_ 

3.1.1 .2 

Determine  Indicators 

.  L. 

3.1. 1.4 

Review  Collection  Plan  File _ _ 

Table  J  8  (Item  16) 
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Task 

NA 

1 

2 

3 

4 

5 

6 

2.1. 2.3 

Post  Collateral  Enemy  0  of  B 

_ 2__ 

,_L_ 

_JL. 

_2_ 

2 

2.1. 2.1 

Determine  Source  ReliabHIty 

JL. 

l 

.  4__ 

_J _ 

2. 1.2.5 

Maint.  Friendly  Sit.  Overlay 

1 

_J_. 

3 

2.1. 2.4 

Maintain  Intelligence  Wrkbook___ 

1 

1 

2 

3.1 .2.1 

Determine  Resource  Available 

1 

1 

1 

1 

2.1.2^ 

Post  Eltmc  0  of  B  Battle  Map 

2 

1 

3.1 .2-2 

Determine  Resource  Capability 

1 

i 

2.1 .2.8 

Maintain  Templating  Files  _ 

“T" 

~r 

'7 - 

3.1 .2.3 

Identify  Other  Sources 

1 

“7 

3.1.1  J2. 

Determine  Indicators.  _ 

~T~ 

3.1. 1.4 

Review  Collection  Plan  File _ 

Tab)*  IS  (Item  17) 


Task 

NA 

1 

2 

3 

4 

5 

6 

2. 1.2.3 

Post  Coilaterai  Enemy  0  of  B_ _ 

? 

.  S.— 

2. 1.2.1 

Determine  Source  Reliability 

.5 

? 

1 

1 

? 

2.1. 2.6 

Maint.  Friendly  Sit.  Overlay 

_ L_ 

2 

_J  . 

1 

2.1. 2.4 

Maintain  Intelligence  Writ  book 

__2_ 

1 

3.1. 2.1 

Determine  Resource  Available 

1 

1 

2 

2.1. 2.2 

Post  Eltmc  0  of  B  Battle  Map 

.  i_ 

3.1  A2 

Determine  Resource  Capability 

1 

2 

2.1. 2.8 

Maintain  Temoleting  Flies _ 

l 

1 

I 

3.1  .2.3 

Identify  Other  Sources 

1 

"1 

3.1.1 .2 

Determine  Indicators 

1 

3.1. 1.4 

Review  Collection  Plan  File _ 

_i_ 

Table  20  (Item  1 8) 

ttams  78.  17,  ana  18  compared  three  different  METHODS  for  recording  the  outcomes 
of  a  task  step,  each  rated  on  a  scale  from  1  --Highly  useful,  to  8— Of  little  use.  The 


methods  in  order  of  presentation  are: 

e  A  displayed  menu  of  possible  outcomes  where  on*  could  check  off  those  that 
occur. 

*  A  fllMn-the-blank  format  for  entering  numerical  values,  dates,  etc. 
e  Free  format  typing  of  notes. 

Preferences  appear  biased  toward  the  greater  flexibility  of  free-form  typing  entries, 
and  perhaps  toward  the  smaller  memory  demands  posed  by  using  menu  lists  for 
selections.  Some  differences  were  discernible  among  tasks  for  these  three  Items. 

-  3-22  - 


m 


For  task  2. 1.2.3— (Post  Collateral  Enemy  Order  of  Battle)  all  entry  inodes  were  rated 
as  useful.  For  task  2. 1.2.5— (Maintain  Friendly  Situation  Overlay)  none  were  rated  as 
useful.  For  task  2. 1.2.1— (Determine  Source  Reliability)  there  was  no  consensus.  For 
task  3.1. 2.1—  (Determine  Resource  Availability)  all  entry  modes  were  rated  as  of  lit¬ 
tle  use.  The  ratings  do  not  appear  to  reflect  discriminations  between  the  different 
modes  for  recording  task  step  outcomes,  but  rathe'  the  usefulness  or  lack  of  useful¬ 
ness  of  recording  step  outcomes  In  any  manner  for  the  particular  task. 

3.6*4  Comments  for  Items  on  Individual  Tasks 

Appendix  D  contains  transcriptions  of  the  comments  which  participants  appended  to 

their  ratings  of  task  step  descriptions.  The  transcriptions  are  presented  in  the 

/ 

sequence  of  task  descriptions  in  the  booklet,  with  comments  for  each  task  presented 
In  the  order  of  the  Items  In  the  response  form.  While  the  comments  are  too  numerous 
for  easy  summary,  examples  are  provided  for  tasks  rated  by  a  number  of  partici¬ 
pants. 

For  task  2. 7 .2. 7  —(Determine  Source  Reliability)  comments  were  numerous  because 
all  participants  used  the  task  for  practice.  Examples  of  comments  from  thirteen  par¬ 
ticipants  are:  ^ 

e  Analyst  experience,  messages  redundancy,  situational  continuity  all  enter  into  the 
making  of  reliability  estimates  for  a  particular  message 

s' 

e  Cheeking  source  reliability  is  often  superfluous  If  sourbe  is  known. 

e  Source  reliability  is  most  important  on  spot-type  reports  which  are  almost  always 
time  critical. 

e  If  there  is  not  enough  time,  eliminate  step  2  (determine  specific  value  of  source) 
of  this  task. 

e  Sometimes  one  must  act  on  the  information  first,  then  complete  processing. 
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•  Your  memory  can  ba  a  substitute  for  a  fils. 

For  task  2. 1 .2.2— ( Post  Electronic  Order  of  Bottle  Map)  there  were  a  number  of 
suggestions  for  changing  the  given  step  sequences  and  the  boundaries  with  other 
tasks.  Other  comments  from  three  participants  were: 

e  This  task  cannot  be  eliminated;  would  cause  deterioration  of  all-source  products. 

a  Manual  methods  for  data  storage/ retrieval  are  very  time-consuming  and  difficult. 

For  task  2.1 .23— (Post  Collateral  Enemy  Order  of  Battle )  there  were  concerted 
suggestions  from  eleven  participants  about  re-ordering  the  steps  of  the  task  as 
given  in  the  descriptive  materials: 

e  Steps  should  be  re-ordered  In  terms  of  Immediate  need/interest, 
e  Steps  as  given  are  out  of  sequence. 

e  Considerations  discussed  for  eliminating  some  task  activities. 

e  Deferr ability  of  this  task  depends  on  the  data-- no  deferment  if  data  are  "event- 
generated". 

For  task  2.1 .2.4— (Maintain  Intelligence  Workbook)  comments  in  relation  to  various 
rating  Items  from  four  participants  included: 

e  Should  be  keyed  against  pre-set  (IPB)  patterns,  doctrinal  templates,  etc. 

e  If  there  Is  no  time,  there  is  little  need  to  process  this  task  once  the  data  are 
posted  to  EOB. 

e  Earlier  step  Information  is  constantly  necessary:  Must  be  able  to  recognize  pat¬ 
terns,  and  determine  when  data  Is  obsolete. 

For  task  2.1 3.6— (Maintain  Friendly  Situation  Overlay)  comments  from  five  partici¬ 
pants  included: 

e  Combining  of  enemy  and  friendly  situation  displays  la  desirable  if  practical;  Intelli- 
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genes  Information  must  be  sanitized  before  posting  to  G-3  map. 
e  Boundary  condition  postings  are  non-deferrable. 

e  If  situation  changes  too  rapidly,  commander  is  briefed  from  memory,  and  process¬ 
ing  is  deferred. 

For  task  2.1 .2.6— (Maintain  Tampiating  File)  comments  from  three  participants 
included: 

e  Not  readily  deferrable,  -other  than  doctrinal  templates-  others  are  more  or  less 
progressive. 

e  For  good  analysis  one  needs  a  base  to  work  from;  IPB  Is  that  base. 

e  Actual  postings  of  templates  Is  very  time-consuming;  event  or  decision  theme 
codes  are  usually  briefed... 

e  Many  things  are  based  on  memory  because  it  is  too  time  consuming  flipping 
through  references  to  verify  templates. 

e  Recall/retrieval  is  important. 

3.5.5  Debriefing  Comments 

When  all  participants  in  a  session  had  completed  their  response  forms  a  debriefing 
discussion  was  held.  Comments  can  be  summarized  as  follows: 

3.5.5. 7  General  reactions  to  session  " 

A  number  of  participants  mentioned  that  they  appreciated  the  chance  to  express 
their  views  and  contribute  to  the  solution  of  problems  they  perceived  as  real  and 
Important.  With  regard  to  being  asked  to  judge  the  Information  needs  of  Intelligenca 
analysts,  several  observed  that  It  Is  hard  to  pre-judge  or  predict  what  will  be 
needed,  even  by  oneself,  without  a  tactical  context.  It  becomes  even  harder  to 
Judge  Information  needs  for  others,  without  knowing  the  Individual’s  situation,  level  of 
task-knowledge,  training,  and  experience,  and  so  on. 
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3 .5.5.2  Reservations  about  Automated  Systems 


A  remark  that  occurred  in  one  form  or  another  In  almost  every  debriefing  session 
was:  "What  happens  if  the  automated  system  breaks  down? --Analysts  must  NOT 
become  helpless  in  such  situations."  Techniques  for  backup  in  the  form  of  machine 
redundancy  and  hardcopy  outputs  were  mentioned.  Other  suggestions  for  system 
failure  were  to  gear  system  inputs,  outputs,  and  alternative  procedures  for  minimum 
conflict  with  initiating  manual  fall-back  procedures,  and  to  minimize  shock  from 
operating  in  isolated  situations  with  communications  that  are  spotty  or  Inoperative. 

3.5.S.3  Field  Safes  and  Foot  locker  Materials 

Comments  in  several  sessions  were  directed  at  the  need  for  storage  of  reference 
materials:  Extensive  possibilities  for  automated  information  services  were  perceived 
here  (some  noted  that  sometimes  six  or  seven  safes  or  lockers  full  of  reference 
materials  may  accompany  a  headquarters  level  intelligence  unit).  While  most  of  this 
information  is  used  infrequently,  access  to  it  can  be  crucial.  There  were  reserva¬ 
tions  about  automating  such  storage  however.  Admittedly,  In  manual  operations  there 
often  is  not  enough  time  to  find  very  much  of  the  pertinent  footlocker  data  that 
applies  to  the  solution  of  an  analytic  problem.  Nevertheless,  some  discussants  could 
not  see  how  such  data  could  be  stored  efficiently  or  easily  accessed  in  an 
automated  system.  Additional  concerns  focused  on  the  user’s  need  to  browse 
through  relevant  areas  and  to  accurately  query  the  data  base.  The  requirement  for 
a  user-oriented  query  language  Is  clsar. 

3 J5.5.4  Intelligence  Workbook 

For  a  number  of  participants  some  form  of  automated  aid  for  the  intelligence  work¬ 
book  functions  seemed  very  desirable.  The  workbook  can  function  in  a  number  of 
ways  in  manual  operations:  as  a  repository  for  reminder  notes;  as  a  source  of  refer¬ 
ences  to  current  summary  data;  to  supply  cross-references  to  display  items,  etc. 


i 
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The  Impression  from  the  comments  was  that  the  intelligence  workbook  is  a  main  aid 
for  supporting  various  aspects  of  analysts’  memories  in  recalling  details  of  recent 
events  of  significance,  especially  during  conditions  of  high  work-toad. 

3. 5. 5.5  Massage  Formats 

A  frequently  repeated  opinion  was  that  message  formats  could  and  ought  to  be  kept 
as  simple  as  practical.  Several  participants  gave  examples  of  message  elements 
they  judged  to  be  of  more  or  less  utility  for  most  situations.  In  essence,  some  ques¬ 
tioned  the  cost/effectiveness  of  retaining  lower  utility  elements  for  most  tactical 
intelligence  purposes.  Besides  date/time  group,  source  type,  and  sender  ID  as 
necessary  items,  there  were  varying  opinions  about  the  importance  of  message  ele¬ 
ment  types  included  in,  for  example,  the  "salute"  reporting  mnemonic  (size,  activity, 
location,  unit,  time,  equipment).  Some  of  the  differences  appeared  to  result  from 
opinions  about  the  relative  importance  of  functions  served  by  the  information  in  mes¬ 
sages,  such  as  indications  and  warning,  targeting,  enemy  situation,  order  of  battle, 
planning,  etc. 

3.5.6.6  Displays 

The  discussion  of  displays  had  several  themes.  Two  of  these  were  retention  and 
proper  handling  of  time  notation  symbology,  and  reduction  of  visual  clutter;  themes 
which  are  interrelated  to  a  degree.  The  flexibility,  field-portability,  and  reliability  of 
the  manual  displays  using  maps,  acetate  overlays,  and  grease  pencils,  also  was 
pointed  out  repeatedly.  Sometimes  there  were  explicit  remarks  that  such  crucial 
advantages  of  the  manual  operations  must  not  be  sacrificed  while  trying  to  gain  the 
desirable  update  ease  and  speed  of  computer-driven  displays.  The  use  of  multiple 
acetate  overlays  to  reduce  clutter  was  mentioned  as  very  valuable  but  cumbersome 
and  time  consuming.  The  use  of  different  colored  grease  pencils  for  different  report¬ 
ing  time  slices  also  was  mentioned  as  very  effective.  Both  these  techniques  were 
pointed  out  as  examples  of  desirable  display  capabilities  that  were  probably  easy  to 
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achieve  with  computer  displays  while  also  offering  rapid  and  easy  update  and 
archival  display  storage  (history)  capabilities.  Several  remarked  that  displays,  like 
message  formats,  must  be  kept  very  simple  to  be  usable. 

3.6  Discussion  of  Implications 

Results  from  the  current  research  phase  revealed  some  patterns  in  the  opinions  of 
intelligence  analysts  regarding  how  they  procesa  information  and  how  an  automated 
system  might  aid  in  the  process.  There  were  systematic  patterns  in  the  variable 
degrees  of  confidence  with  which  the  participants  answered  the  questions  about 
intelligence  processing  tasks.  Participants  were  able  to  comment  about  the  compara¬ 
tive  usefulness  and  boundary  appropriateness  for  task  descriptions.  They  had  sys¬ 
tematic  opinions  about  the  eiiminabillty,  deferrability,  demandingness,  interdepen¬ 
dence,  and  information  prerequisites  of  tasks.  They  discriminated  among  the  useful¬ 
ness  values  of  task  checkoff  and  cueing  aids  for  different  tasks.  They  had  focused 
concerns  and  reservations  about  aspects  of  automation  for  tactical  intelligence  pro¬ 
cessing.  They  discriminated  between  automation  support  for  different  types  of  infor¬ 
mation  such  as  that  presently  stored  in  safes,  footlockers,  and  workbooks  or  jour¬ 
nals.  They  had  clearly  formulated  opinions  about  simplification  of  message  formats 
and  tactical  information  displays.  Such  considerations  provide  initial  indications 
about  what  query  methods  have  potential  as  aids  for  Intelligence  analysts. 

3.6.0.7  Study  Design  Results  should  be  accepted  only  as  preliminary,  taking  several 
factors  into  account.  On  the  plus  side,  participants  were  experts  on  a  variety  of 
Intelligence  topics  and  shared  their  knowledge  without  hesitation.  They  were  pro¬ 
vided  with  specific  task  descriptions  and  rating  items  that  should  have  minimized  the 
effects  of  ambiguity  on  results.  However,  two  factors  of  the  research  may  have 
lead  to  rather  conservative  data.  First  of  all,  the  participants  had  no  introduction  to 
the  research  or  specific  review  before  coming  to  the  session.  Even  one  evening  of 
"boning  up"  would  probably  have  tightened  the  pattern  of  results  obtained. 
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Secondly,  participants  were  not  allowed  to  confer  during  the  rating  work  and  share 
ideas  about  the  problems.  This  eliminated  the  possibility  of  deepening  and  refining  a 
common  frame  of  reference.  Consequently,  data  reflect  knowledgeable  opinions 
where  the  likelihood  of  consensus  was  lower  that  it  might  have  been  under  less  con¬ 
trolled  conditions.  Perhaps,  the  spontaneity  of  the  research  situation  helped  to 
guarantee  realistic  results. 

Despite  the  limitations  just  cited,  perceptibly  tight  CLUSTERS,  at  or  near  opinion 
EXTREMES  were  obtained  for  one  or  more  of  the  tasks  for  ratings  on  twelve  of  the 
eighteen  Items.  The  clusters  were  indicative  of  some  consensus  on  discriminations 
between  tasks  with  respect  to  preferences  for  ways  to  cope  with  task  load, 
demandingness  of  various  tasks,  dependencies  between  task  steps,  and  the  use  of 
automated  aids  for  task  results  checkoff. 


f 
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4.  EVALUATION  OF  A  FAMILIARIZATION/REFRESHER  DISPLAY  IN  MIQSTURE 

4.1  Introduction 

The  purpose  of  this  (second)  project  for  evaluation  of  MIQSTURE  was  to  test  a  CRT 
display  arrangement  for  Its  value  in  familiarizing  and/or  refreshing  the  user/operator 
with  infrequently  used  but  essential  elements  of  the  interaction  language.  In  this 
arrangement,  language  elements  entered  by  the  user/operator  in  a  familiar  ("easy 
entry")  format  are  transformed  and  immediately  displayed  in  a  less  frequently  used 
format  comprising  a  more  flexible  data  entry  procedure.  The  intent  In  providing  the 
familiarization/refresher  display  is  to  maintain  an  immediately  useful  level  of  familiar¬ 
ity  for  the  user/operator  for  the  more  flexible  but  less  used  entry  format. 

/ 

4.2  Problem  Background 

The  user/operator  would  interact  with  the  proposed  MIQSTURE  system  via  a  work 
station  consisting  of  a  TV-like  CRT  display,  a  standard  typewriter  keyboard,  and 
groups  or  "pads"  of  function  keys.  User/operator  actions  may  involve  many  different 
combinations  of:  single  function-key  actions:  positioning  of  a  movable  cursor  spot  in 
the  display  followed  by  a  function-key  or  keyboard  action;  typing  in  single  charac- 
ters  or  strings  of  characters;  and  so  on.  To  accomplish  one  transaction  step  in  an 
interactive  dialog  with  the  system,  a  combination  of  such  actions  is  often  neces¬ 
sary. 

Such  transaction  steps  in  the  user/operator-system  dialog  vary  on  two  dimensions: 

1 .  The  overall  frequency  of  occurrence  of  the  step  in  dialog. 

2.  The  variability  of  actions  allowed  or  required  in  the  step. 

An  example  of  a  transaction  step  showing  high  frequency  of  occurrence  and  high  uni¬ 
formity  of  actions  is  the  "EXECUTE"  command  in  MIQSTURE.  This  step  is  performed 
without  variation  many  times  for  many  purposes  during  most  interaction  dialogs.  At 
the  other  extreme,  a  class  of  Interaction  steps  with  low  uniformity  of  actions  and  low 
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frequencies  of  occurrence  are  the  query  formulations  used  to  search  for  statistically 
rare  or  unique  topical  combinations. 


4.3  Problem  Statement 

Several  considerations  entered  into  framing  the  research  problem: 

e  High-frequency,  high  uniformity  transaction  steps  often  involve  easy  initial  learn¬ 
ing  and  later  skill  maintenance. 

a  Low-frequency,  low  uniformity  transaction  steps  often  involve  difficult  initial 
learning  as  well  as  difficult  later  maintenance  of  familiarity. 

e  The  difficulties  in  the  low  uniformity  transaction  steps  are  usually  inherent  in  the 
steps  themselves,  and  for  the  most  part  cannot  be  eliminated  by  resolving  the 
transaction  steps  into  simpler  ones.  Furthermore,  the  functions  performed  by  the 
complex,  infrequent  transaction  steps  usually  cannot  be  Ignored  or  eliminated. 

e  The  low  frequency  of  use  of  the  more  complex  interaction  steps  means  less  prac¬ 
tice  with  them,  resulting  in  lower  levels  of  familiarity.  The  result  is  often  slower 
performance  and/or  Increased  errors  when  the  complex  steps  must  be  employed 
for  the  first  time  after  a  period  of  disuse. 

S 

A  potentially  successful  interactive  language  design  needs  an  approach  for  increas¬ 
ing  exposure  and  practice  on  complex,  infrequently  used  interaction  steps.  The 
present  research  focused  on  the  possible  value  of  a  familiarization/refresher  display 
arrangement,  whereby  the  more  demanding  formats  for  language  entry  are  displayed 
concurrently  with  the  corresponding  versions  of  less  demanding  formats  actually 
being  used  in  the  transaction  steps.  The  question  Is:  esn  such  an  arrangement  help 
refresh  (improve)  the  user/operator’s  level  of  familiarity  for  immediate  use  of  the 
less  frequently  needed  formats? 
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■eh  Approach 


For  the  experiment,  participants  were  required  to  enter  query  statements  using  two 
different  formats.  The  less  demanding  form  of  entry  was  a  type-ln-the-blank  format 
(see  Figure  4-1),  while  the  more  demanding  format  consisted  of  an  unaided  free- 
key-ln  format  (see  Participant  Orientation  Booklet,  Appendix  F). 


name 
(name) 
nes,  B 

address 

(addr) 

12345  Main  _ 

city 

(city) 

Encino _ 

state 

(stat) 

Ca 

mm 

dept 

(dept) 

09S 

12 -AN 

15-AN 

10-A 

■EH 

5-M 

3-M 

4-M 

Figure  4-1.  Example  of  Type-ln-The-Blank  Format. 


The  project  approach  was  in  two  phases: 


1.  Design  and  develop  the  MIQSTURE  simulation  capability  and  experimental  materi¬ 


als. 


2.  Conduct  experimental  sessions  with  participants. 


Method 


The  MIQSTURE  simulation  capability  consists  of  an  interactive  program  running  on  a 
PDP  1 1  /70  computer  under  the  UNIX  operating  system.  Several  small  data  bases 
were  developed  and  studied  for  the  experiment.  These  included  an  Army  tactical 
intelligence  message  file,  and  a  personnel  records  file  containing  names,  depart¬ 
ments,  salaries,  and  other  data  about  Individual  personnel.  Junior  college  level  pertf- 
cipants  used  in  the  experiment  found  the  contents  of  the  personnel  records  file  to 
be  the  most  familiar  to  their  experience,  so  that  file  was  used  for  the  experiment. 
Experimental  materials  developed  included  a  MIQSTURE  Experimenter’s  Guide 
(Appendix  E)  a  Participant  Orientation  Booklet  (Appendix  F),  and  a  test  query  set 
(Appendix  G). 


The  experimental  session  for  each  participant  consisted  of  seven  steps  (see  Appen¬ 
dix  E  for  details): 

1.  The  greeting,  which  Included  Introductions,  informal  verbal  explanation  of  the 
purposes  and  nature  of  the  experiment,  and  a  brief  tour  of  the  computer  facility. 

2.  The  orientation,  in  which  the  participant  read  the  printed  orientation  materials 
and  mod  in  example  queries  In  the  booklet  (Appendix  F),  and  Included  a  question 
and  answer  session  to  assure  understanding  of  all  points. 

3.  Practice,  performed  by  ait  participants,  in  which  the  participant  was  Introduced 
to  the  terminal  and  provided  with  hands-on  practice  and  coaching,  using  two 
question  items  on  slips  of  paper.  The  practice  consisted  first  of  formulating 
queries  from  the  question  slips  and  entering  each  query  by  typing  in  all  charac¬ 
ters  (data  values,  element  qualifiers,  connectives)  of  the  query  string.  Then  the 
same  queries  ware  re-entered  again,  this  time  by  typing  only  data  values  into 
designated  blanks  in  the  MIQSTURE  fill-in  display  format  which  has  built-in  qual¬ 
ifiers  and  connectives  (see  Appendix  F). 

4.  The  control  phase,  performed  by  all  participants,  in  which  a  standard  sequence 
of  five  question  slips  was  provided,  one  at  a  time.  Each  was  formulated  by  the 
participant  into  a  query  and  entered  into  the  terminal  via  the  fill-in  display  for¬ 
mat. 

6.  The  treatments  phase,  In  which  a  fixed  sequence  of  five  (different)  question 
slips  was  provided,  and  queries  were  entered  via  the  fill-in  format  under  one  of 
three  conditions  for  each  of  three  treatment  groups  of  participants: 

a.  Same  conditions  as  the  preceding  control  phase,  in  which  the  interaction 
history  window  of  the  fill-in  display  (see  Appendix  H)  was  turned  off. 

b.  As  each  data  value,  element  qualifier,  and  query  connective  was  entered  by 
the  user/operator  via  the  fill-in  format,  it  was  immediately  regenerated  and 
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displayed  through  the  interaction  history  window,  in  the  tree-style  key-in 
entry  format. 


c.  The  participant  entered  the  query  via  the  fill-in  format,  during  which  time  the 
interaction  history  window  remained  blank  until  the  participant  took  the 
EXECUTE  action  to  cause  the  query  to  be  used  to  search  the  data  base.  At 
that  point,  the  entire  query  was  displayed  in  the  free-styie  key-ln  format  in 
the  interaction  history  window. 

6.  The  criterion  phase,  again  performed  by  all  participants,  in  which  a  standard 
sequence  of  another  five  question  slips  was  provided,  one  at  a  time,  and  formu¬ 
lated  and  entered  via  the  free-style  key-in  format  (see  Appendix  G  for  a  listing 
of  the  questions). 

7.  The  debriefing  step,  In  which  the  participant  was  thanked  and  compensated  for 
participation,  and  offered  an  opportunity  to  be  briefed  on  the  results  at  a  later 
date. 

4.6  Results 

The  data  analysis  consisted^ of  an  analysis  of  variance  (anova)  performed  on  partici¬ 
pant  performance  times  for  all  three  performance  steps  of  the  experimental  session 
sequence.  Each  performance  time  was  recorded  automatically  in  two  parts: 

1 .  The  think  time,  defined  as  the  interval  between  when  the  participant  received  a 
question  slip  and  made  the  first  query  entry  action  at  the  terminal,  and 

2.  the  key-ln  time,  defined  as  the  interval  between  the  first  entry  action  and  the 
EXECUTE  action,  which  sends  the  finished  query  for  processing. 

In  addition,  a  visual  analysis  of  typographical  errors  was  performed  on  the  automati¬ 
cally  recorded  Interaction  protocols. 
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Figure  4-2  shows  results  of  the  experiment.  The  left-hand  columns  of  the  figure 
provide  response-time  scales  in  seconds  for  think  times  and  key-in  times.  Names  for 
the  three  phases  of  the  experiment  are  arrayed  across  the  top.  Query  numbers  are 
Indicated  at  the  bottom  of  the  column  lines.  Queries  1  through  5  comprise  the  control 
phase,  6  through  1 0  the  treatments  phase,  and  1 1  through  1 5  the  criterion  phase. 
Each  plot  point  represents  the  mean  of  values  for  the  ^>even  participants  in  a  partic¬ 
ular  treatment.  The  treatment  1 .  (No  refresh  display)  plot  points  are  solid  dots;  treat¬ 
ment  2.  (Element  refresh  display)  points  are  Xs,  and  treatment  3.  (Whole  query 
refresh  display)  points  are  circled  dots.  Think-time  points  are  connected  by  solid 
lines,  key-in-time  points  by  dashed  lines. 

Visual  Inspection  of  the  plots  of  the  three  treatment  groups  in  the  control  phase 
shows  little  scatter  and  a  high  correlation  between  key-in-time  means  (dashed  lines) 
across  Items  1  through  6,  and  somewhat  more  scatter  and  lower  correlation  for 
think-time  means  (solid  lines).  An  anova  (21  cases)  for  the  three  treatment  groups  in 
the  control  phase  showed  no  significant  differences,  indicating  that  the  randomiza¬ 
tion  of  participants  was  successful,  .  Similarly,  an  anova  for  overall  treatment  group 

results  for  the  criterion  phase  were  also  non-significant. 

-/■ 

Although  the  overall  treatment  group  differences  for  the  criterion  phase  were  not 
statistically  significant,  further  inspection  of  the  plotted  data  points  provides  some 
interesting  trends: 

1.  There  is  a  trend  for  reduced  mean  think  times  from  left  to  right  across  the  three 
treatments  (an  apparent  learning  effect);  the  overall  difference  between  control 
and  criterion  phase  think  times  is  significant  beyond  the  .01  level  of  confidenco. 

2.  There  is  a  trend  for  increased  key-in  times  between  the  control  and  criterion 
phases,  probably  reflecting  the  increased  keystrokes  required  in  the  free-key-in 
tasks  of  the  criterion  phase  as  compared  to  the  fewer  keystrokes  required  for 
the  fill-in  formats  of  the  control  and  treatments  phases.  The  overall  difference 
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between  the  control  end  criterion  phase  key-in  times  is  statistically  highly  signi¬ 
ficant  (well  beyond  the  .001  level  of  confidence).  A  corollary  of  this  observation 
is  that  there  were  typographical  errors  In  7%  of  the  control  phase  responses,  5% 
jf  the  treatments  phase  responses,  and  26%  of  the  criterion  phase  responses. 
The  overall  difference  in  such  errors  between  the  combined  control  and  treat¬ 
ments  phases  and  the  criterion  phase  Is  also  significant  well  beyond  the  .001 
level  of  confidence.  This  combined  Increase  in  errors  and  increase  in  key-in 
times  represents  a  reliable  difference  In  difficulty  levels  between  the  fill— in  and 
free-key-in  entry  formats. 

3.  There  is  a  suggestion  of  a  “phase-change"  effect  of  increased  response  times 
for  both  think  and  key-in  for  iteais  6  and  1 1 ,  (the  Items  introducing  new  treat-, 
ments  to  the  participants).  This  possible  "distraction*  affect  for  Item  6  of  the 
treatments  phase  appears  most  marked  for  the  key-in  times  of  the  element 
refresh  display  treatment,  next  most  marked  for  the  whole  query  refresh  display 
treatment,  and  least  for  the  no  refresh  display  treatment.  The  possible  distrac¬ 
tion  effect  appears  quite  marked  between  items  1 0  and  1 1  for  both  think  and 
key-in  times  (Item  1 0  being  the  last  flll-ln  task,  and  item  1 1  the  first  free-key-in 
task  for  all  participants). 

4.  While  not  significant,  the  differences  in  think-time  means  for  item  1 1  are 
nevertheless  in  the  expected  (and  desired)  directions  for  the  three  experimen¬ 
tal  treatments.  That  is,  the  no  refresh  display  (control)  treatment  shows  the 
greatest  Increase  in  think  time,  the  element  refresh  display  treatment  shows  an 
Intermediate  Increase,  and  the  whole  query  refresh  display  shows  the  least 
Increase  in  think  time.  This  suggests  that  the  whole  query  refresh  display  may 
have  operated  more  strongly  than  the  others  to  maintain  the  participant’s 
unpracticed  familiarity  with  the  free-key-ln  format  prior  to  Item  1 1 .  However,  it  is 
also  apparent  that  the  familiarization  affect,  if  reliable,  was  nevertheless  largely 
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compensated  by  the  single  practice  trial  provided  by  Item  1 1  itself,  in  view  of 
the  immediate  reduction  of  the  mean  think  times  for  Items  1 2  through  1 6. 

4.0  Discussion 

The  results  clearly  demonstrate  the  greater  demand  on  users  and  the  increase  in 
errors  due  to  free-key-ln  types  of  entry  formats  compared  to  fill-in  types  of  formats. 
Of  course,  requirements  for  the  increased  flexibility  afforded  by  the  more  demanding 
free-key*ln  format  will  vary,  depending  upon  the  particular  tactical  data  processing 
application.  For  some  of  the  applications  requiring  free-key-in,  a  significant  propor¬ 
tion  of  the  entries  may  nevertheless  be  amenable  to  the  easier  fill-in  entry  formats. 
If  this  proportion  is  high  enough,  the  argument  becomes  Strong  for  providing  a  dual¬ 
level  entry  format  capability,  such  as  the  one  available  in  MIQSTURE.  For  such  appli¬ 
cations  (requiring  a  dual-level  entry  language)  the  problem  of  refamfliarizing  the 
user/operator  with  the  rarely  used  but  crucially  flexible  free-key-in  formats  may  be 
a  serious  one. 

While  statistically  equivocal,  the  pattern  of  results  nevertheless  suggest  a  positive 
value  for  familiarize/refresh  display  types  of  arrangements  in  coping  with  use  of 
free-key-in  formats.  Despite  the  limited  sample  of  participants,  the  pattern  of 
results  obtained  are  consistent  with  expectations,  and  provide  encouragement  for 
pursuing  this  approach  to  interactive  language  display  and  utilization. 
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6.  Conclusion 


The  current  report  provides  designers  of  automated  systems  with  preliminary  infor¬ 
mation  about  selected  ways  to  structure  user-computer  dialogues  for  tactical  intelli¬ 
gence  applications.  In  designing  an  interactive  query  language,  both  the  importance 
of  a  task  and  characteristics  of  its  components  must  be  taken  into  account;  cues 
and  prompts  are  not  equally  helpful  in  all  types  of  intelligence  tasks  for  reminding 
users  about  correct  input  procedures.  Overall,  the  research  data  provide  insights  for 
the  designers  of  such  languages  and  indicate  that  an  interactive  query  language 
approach  to  tactical  intelligence  is  quite  promising. 
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6.  APPENDICES 


A -Task  Description  Booklet 

B-MIQSTURE  interview  Guide 

C-Response  Form  Rating  Distributions  by  Tasks 

O-Transcription  of  Comments 

E-MIQSTURE  Experimenter's  Guide 

F-Participant  Orientation  Booklet 

G-Test  Questions  Set 
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A- Task  Description  Booklet 


2.0  ALL  SOURCE  PRODUCTION 

2.1  DATA  BASE  MANAGEMENT 

2.1.1  MAINTAIN  INTELLIGENCE  FILES 

2. 1.2.1  DETERMINE  SOURCE  RELIABILITY 

2. 1.2.1  DETERMINE  SOURCE  RELIABILITY 

1.  REVIEW  REPORT  TO  DETERMINE  THE 

1.  CHECK  RELIABILITY  (A-F)  AND 

SOURCE  OF  THE  INTELLIGENCE 

ACCURACY  (1-6)  CODES  FOR 

INFORMATION.  (E.G.,  UNIT, 

REPORT.  IF  NOT  ASSIGNED. 

INDIVIDUAL,  SENSOR,  ETC). 

EXAMINE  SOURCE  DESIGNATION 

FORMAT  ELEMENTS  OF  REPORT. 

2.  CHECK  THE  SOURCE  RELIABILITY 

FILE  AND  DETERMINE  THE  ALPHA- 

2.  RETRIEVE  SOURCE  RELIABILITY 

BETIC  VALUE  FOR  THE  SOURCE  OF 

DISPLAY  FOR  SOURCE  DESIGNA- 

THE  INTELLIGENCE  REPORT  AND 

'TION  IN  REPORT. 

ANNOTATE  THE  REPORT. 

3.  ASSIGN  RELIABILITY  AND 

ACCURACY  ESTIMATES  FOR  SOURCE 

FROM  SOURCE  RELIABILITY  FILE. 

2. 1.2.2  POST  ELECTRONIC  ENEMY  ORDER 

2. 1.2. 2  POST  ELECTRONIC  ENEMY  ORDER 

OF  BATTLE  MAP 

OF  BATTLE  MAP 

EXTRACT  FROM  THE  SIGNALS 

1.  ON  DUAL  SCREENS: 

INTELLIGENCE  REPORT  AND  POST  TO 

EEOB  MAP  THE  FOLLOWING  DATA: 

a.  REVIEW  SIGINT  REPORT  FORMAT 

(4  ELEMENTS). 

A.  EMITTER  TY«E 

b.  REVIEW  PROVISIONAL 

B.  UNIT  IDENTIFICATION  OR  LEVEL 

AUTOMATIC  EEOB  POSTING 

OF  COMMAND 

ENTRY  (BLINKERED  ON  EEOB 

UPDATE/MAINTAIN  DISPLAY). 

C.  LOCATION  AND  TIME 

c.  MAKE  ADJUSTMENTS  IF 

D.  TYPE  WEAPON  SYSTEM 

NECESSARY. 

d.  RELEASE  PROVISIONAL  EEOB 

|S IGINT  REPORT| 

POSTING  TO  ACTIVE  EEOB 

DISPLAY. 

IEEOB  MAPI 

2. 1.2.3  POST  COLLATERAL  ENEMY  ORDER  OF 
BATTLE 

EXTRACT  FROM  THE  INTELLIGENCE 
REPORT  AND  POST  TO  THE  EOB 
MAP  THE  FOLLOWING  DATA: 

a.  UNIT  DESIGNATION  AND 
IDENTIFICATION 

b.  LOCATION  AND  DISPOSITION 

c.  STRENGTH  AND  COMPOSITION 

d.  TIME  AND  EVENT 

e.  WEAPON  SYSTEM 


2. 1.2.4  MAINTAIN  INTELLIGENCE  WORKBOOK 

1.  REVIEW  EACH  INTELLIGENCE  REPORT 
FOR  KEY  INFO. 

2.  POST  INFORMATION  TO  APPROPRIATE 
SECTION  OF  THE  INTELLIGENCE 
WORKBOOK  (EXAMPLES) 


a. 

INTEL 

i . 

a 

MOVEMENT 

b. 

EW 

j. 

^STRENGTH  h 

c. 

INF 

k. 

CAPABILITIES 

d. 

ARMOR 

1. 

VULNERABILITIES 

e. 

ARTY 

m. 

EQUIPMENT 

f. 

AD 

n. 

PERSONALITIES 

g- 

AIR 

0. 

CBR 

h. 

ENG 

P- 

CONCLUSIONS 

UPDATE  THE  WORKBOOK  AS  MORE  CUR¬ 
RENT  DATA  BECOMES  AVAILABLE. 
DELETE  OBSOLETE  DATA. 

INTELLIGENCE  REPORT! 


lEOB  MAPI 

INTELLIGENCE  WORKBOQKl 


2. 1.2.3  POST  COLLATORAL  ENEMY  ORDER 

OF  BATTLE 

a.  FROM  CRT  DISPLAYED  INTREP, 
USE  TEXT-EDIT 
CAPABILITIES  TO  COPY-OUT, 
OR  KEY-IN,  5  INFO 

ETements. 

b.  REVIEW  EXTRACTED/ KEYED 
ELEMENTS  FOR  CORRECTNESS. 

c.  CALL  EOB  UPDATE/MAINTAIN 
DISPLAY  ON  ADJOINING 
SCREEN. 

d.  USING  EOB  DISPLAY  AND 
REPORT  CURSORS  AND 
"TRANSFER"  BUTTON,  POST 
5  ELEMENTS  TO  EOB  U/M 
DISPLAY,  AND  REVIEW. 

e.  TAKE  "RELEASE"  ACTION  TO 
RELEASE  PROVISIONAL 
POSITIONS  TO  ACTIVE  EOB 
DISPLAY. 

2. 1.2.4  MAINTAIN  INTELLIGENCE  WORKBOOK 

a.  REVIEW  CRT -DISPLAYED 
INTREP  FOR  KEY  INFO. 

b.  FOR  EACH  KEY  CATEGORY 
(A.  TRROUGH  P.) : 

1.  TEXT-EDIT  EXTRACT  OR 
KEY- IN  REPORT  DATATO 
"HOLD"  AREA  ON  REPORT 
DISPLAY  CRT. 

2.  CALL  APPROPRIATE  AREA 
OF  WORKBOOK  FILE  DIS¬ 
PLAY  ON  ADJOINING  CRT 
DISPLAY. 

3.  USING  CURSORS  ON  BOTH 
SCREENS,  AND  "TRANSFER' 
BUTTON  TO  POST  THE 
WORKBOOK  FILE. 

4.  TAKE  "RELEASE  ACTION  TO 
MOVE  NEW  POSTINGS  TO 
ACTIVE  WORKBOOD  FILE. 


2. 1.2. 5  MAINTAIN  FRIENDLY  SITUATION 

OVERLAY. 

1. 

PERIODICALLY  POST  CURRENT 

FRIENDLY  UNIT  DISPOSITIONS, 

AS  CARRIED  ON  THE  G3  SITMAP, 

ONTO  AN  OVERLAY  FOR  USE  BY  ALL 
ELEMENTS  OF  THE  EWIOC. 

2. 

REVIEW  INTELLIGENCE/EW  REPORTS 

FOR  DISPOSITION  OF  ASSETS  AND 

POST  TO  FRENSIT  OVERLAY. 

2. 1.2.6  MAINTAIN  TEMPLATING  FILES 

ESTABLISH  AND  MAINTAIN  TEMPLATING 
FILES  CONSISTING  OF: 

a. 

DOCTRINAL  TEMPLATES 

b. 

SITUATION  TEMPLATES 

c. 

EVENT  TEMPLATES 

d. 

DECISION  TEMPLATES 

GS  SITMAP 


INTEL/EW  REPORTS 


FRIENDLY  SITUATION  OV 


TUATION  TEMPLATE  FILE 


SION  TEMPLATE  FILE 


TEMPLATING  FILES 


ITlSff 


2. 1.2. 5  MAINTAIN  FRIENDLY  SITUATION 


1.  G3  FRENSIT  OVERLAY  IS  AUTO¬ 
MATICALLY  AVAILABLE  TO  ALL 
EWIOC  SITUATION  DISPLAYS. 

2.  (STEPS  ARE  ANALOGOUS  TO 
THOSE  FOR  2. 1.2.3). 

2. 1.2.6  MAINTAIN  TEMPLATING  FILES 

FOR  EACH  TEMPLATE  TYPE,  PERFORM 

PROVISIONAL  MODIFICATION  WORKUP: 

£.  ON  CONTINUING  BASIS,  ASSIGN 
TEMPORARY  CROSS  REFERENCE 
LINKS  BETWEEN  INCOMING  RE¬ 
PORT  NO.  AND  EXISTING 
TEMPLATE  NOS.  FOR  THE  PRE¬ 
SCRIBED  PERIOD  OF  A  MODIFI¬ 
CATION  WORKUP  CYCLE,  INPUT 
PHASE. 

b.  AT  CLOSE  OF  INPUT  PHASE  FOR 

EACH  X-REFERED  TEMPLATE, 
REVIEW  ALL  REPORTS  Z-REFER- 
ENCED  TO  THAT  TEMPLATE  (ON  i 
CRT).  ! 

i 

c.  ADJUST  TEMPLATE  IN  LIGHT  OF; 
RELEVANT  INFO.  IN  REPORTS. 

d.  ADJUST  TEMPORARY  X-REFER-  : 

ENCES  TO  MATCH  WORKUP  DE¬ 
CISIONS.  | 

e.  RELEASE  ADJUSTED  TEMPLATE 
AND  ADJUSTED  X-REFERENCED  ; 
TO  WORKUP  REVIEW  PROCESS. 
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3.0  INTELLIGENCE  COLLECTION  MANAGEMENT 


3.1  INTELLIGENCE  COLLECTION  PLANNING 


3.1.2  DETERMINE  RESOURCE  AVAIL/CAP 


3. 1.2.1  DETERMINE  RESOURCE  AVAILABILITY 

1.  DETERMINE  THE  RESOURCE  ALLOCATION 
LIST  AND  DETERMINE  THE  AVAIL¬ 
ABILITY  OF  COLLECTION  RESOURCES 
TO  MEET  THE  REQUIRED  COLLECTION 
TASKS. 

2.  CHECK  THE  EQUIPMENT  STATUS  REPORTS 
AND  DETERMINE  THE  OPERATIONAL 
STATUS  OF  THE  ALLOCATED  RESOURCES. 

3.  COMPILE  A  LISTING  OF  AVAILABLE 
RESOURCES  TO  SUPPORT  THE  INTEL¬ 
LIGENCE  COLLECTION  TASKS. 


fCQLLECTION  PLANNING  FILEl 
[RESOURCE  ALLOCATION  LISTj 

EQUIPMENT  STATUS  REPORTS 
RESOURCE  AVAILABILITY  LIST 


3. 1.2.1  DETERMINE  RESOURCE  AVAIL/CAP 

1.  CALL  UP  MASTER  RESOURCE 
ALLOCATION  DISPLAY, 
SKIP/SCROLL  TO  RESOURCES 
AVAILABLE  FOR  REQUIRED 
COLLECTION  TASKS. 

EXAMINE  CURRENT  OPERA¬ 
TIONAL  STATUS  INFORMATION 
ASSOCIATED  WITH  EACH 

'  RESOURCE.  FOR  EACH 

RESOURCE  THAT  MEETS 
JOINT  ALLOCATION/OPERA¬ 
TIONAL  READINESS  AVAILA¬ 
BILITY,  TAKE  "COPYOVER'1 
ACTION  TO  COLLECTION 
PLANNING  LIST  ON  ADJA¬ 
CENT  CRT  SCREEN. 

2.  REVIEW  PLANNING  LIST  AND 
RELEASE  TO  WORK- IN¬ 
PROGRESS  HOLD  FILE. 


COLLECTION  PLANNING  FILES! 


« 


3. 1.2.2  DETERMINE  RESOURCE  CAPABILITY 


1.  REVIEW  THE  COLLECTION  PLANNING 
FILES  AND  DETERMINE  THE  CAPABIL¬ 
ITY  OF  THE  AVAILABLE  RESOURCES 
TO  SATISFY  THE  COLLECTION  TASKS. 


2.  DEVELOP  A  LISTING  OF  RESOURCES 
WHICH  ARE  AVAILABLE  AND  CAPABLE 
OF  FULFILLING  SPECIFIC  EEI/OIR 
INFORMATION  NEEDS. 


NOTE:  RESOURCE  CAPABILITY  LIST 
IS  PROVIDED  BY  THE  COL¬ 
LECTION  OPERATING  ELEMENT 
TO  THE  MISSION  MANAGEMENT 
ELEMENT. 


3. 1.2. 2  DETERMINE  RESOURCE  CAPABILITY 

1.  PLACE  REVIEWED  RESOURCE 
PLANNING  LIST  FILE  ON  ONE 
CRT,  PUCE  SPECIFIC  IN¬ 
FORMATION  ITEMS  LIST  ON 
ADJACENT  CRT. 

2.  SCROLL  THRU  INFO  ITEMS 
LIST.  FOR  EACH  ITEM: 

3.  SCROLL  THRU  RESOURCE 
PLANNING  LIST.  FOR  EACH 
RESOURCE,  ASSIGN  ONE  OF 
3  VALUES  FOR  USE  TO 
COLLECT  INFO  ITEM: 

/ 

a.  PREFERRED 

b.  POSSIBLE 

c.  UST  RESORT 

4.  WHEN  FINISHED  WITH  ALL 
INFO.  ITEMS,  RELEASE 
RATED  ITEM  LIST  TO  MISSION 
WORK  FILE. 

5.  RESULTS  ARE  INVERTED  AND 
WRITTEN  TO  MWF  ORGANIZED 
BY: 

a.  COLLECTION  RESOURCE 

b.  PRIORITY  OF  SPECIFIC 
INFO.  ITEM 

6.  ADJUST  MWF  USING  TEXT-EDIT 
ELIMINATING  DUPLICATES  BY 
RUNNING  AUTOMATIC  DUPLI¬ 
CATE  CHECK  WHICH  HIGH¬ 
LIGHTS  ITEM  APPEARING 
UNDER  MORE  THAN  ONE 
RESOURCE. 

7.  REVIEW  FOR  CORRECTNESS, 
RELEASE  TO  MISSION  PREP 
SECTOR  OF  EEI/OIR  FILE. 


Mfem- 


3. 1.2.3 


IDENTIFY  OTHER  SOURCES 


3.1 


1.  REVIEW  THE  EEI/OIR  FILE  AND 
IDENTIFY  THE  SPECIFIC  INFORM¬ 
ATION  REQUIREMENTS  WHICH  CAN¬ 
NOT  BE  MET  BY  CEWI  GROUP 
RESOURCES. 


2.  REVIEW  THE  CAPABILITY  OF  OTHER 
SOURCES  TO  FULFILL  THE  SPECIFIC 
INFORMATION  REQUIREMENTS. 


3.  DEVELOP  A  LIST  OF  THOSE  OTHER 
SOURCES  AND  PUT  IT  IN  THE  COL¬ 
LECTION  PLANNING  FILE 

a.  DIVISIONS  AND  SEPARATE  BRIGADES 
AND  REGIMENT. 

b.  OTHER  CORPS  RESOURCES 

c.  ADJACENT  CORPS  RESOURCES 

d.  ECHELON  ABOVE  CORPS,  INTER 
SERVICE  AND  ALLIED  NATION 
RESOURCES. 

e.  NATIONAL  AND  StRATEGIC 
RESOURCES 
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.2.3  IDENTIFY  OTHER  SOURCES 
(ANALOGOUS  TO  STEP  1  OF 
3. 1.2.1) 


/ 
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3.1. 1.2  DETERMINE  INDICATORS 

1. 

REVIEW  THE  EEI/OIR  AND  THE 

ESTIMATED  ENEMY  SITUATION  TO 

SELECT  THE  APPROPRIATE  EVENT 

ANALYSIS  MATRICES.  IF  NO 

EVENT  ANALYSIS  MATRIX  IS 

AVAILABLE  FOR  A  PARTICULAR 

EEI/OIR  REQUEST  ASP  DEVELOP 

THE  REQUIRED  MATRIX. 

2. 

EXAMINE  THE  EVENT  ANALYSIS  MATRICES 
TO  DETERMINE  THE  INDICATORS  FOR 

EACH  EEI/OIR. 

3. 

ADD  PRIORITY  ASSOCIATED  WITH 

EACH  EEI/OIR  TO  ITS  INDICATORS. 

4. 

CHECK  FOR  AND  CROSS-REFERENCE 
DUPLICATE  ENTRIES  TO  ELIMINATE 
DUPLICATE  TASKING  AND  PROCESSING 

AND  ADD  INDICATORS  TO  THE  EEI/OIR 
FILE. 

RSuSIHHii 


MY  SITUATION 


3. 1.1.2  DETERMINE  INDICATORS 

1.  CALL  HARDCOPY  PRINT  OF 
ESTIMATED  ENEMY  SITUATION 
AND  SCROLLABLE  CRT  MAIN¬ 
TENANCE  DISPLAY  OF  EEI/ 
OIR  IN  CONTEXT  OF  EES, 
AND: 

a.  SCROLL  THROUGH: EVENT 
ANALYSIS  MATRIX 
CATALOG  DISPLAY  ON 
AbJOINING  SCREEN, 
SELECT  RELEVANT 
MATRICES. 

b.  FROM  EACH  SELECTED 
MATRIX,  IDENTIFY 
APPROPRIATE  INDICA¬ 
TORS. 

C.  FOR  EACH  APPROPRIATE 
INDICATOR,  USE 
CUSORSONBOTH  SCREENS 
AND  "COPY  OUT”  BUTTON 
TO  APPEND  TO  EEI/OIR 
ENTRY.  WHEN  FINISHE 
SIGNAL. 

d.  SYSTEM  AUTOMATICALLY 
ORDERS  INDICATOR  BY 
ASSOCIATED  EEI/OIR 
PRIORITIES,  AND 
BUILDS  INDICATORS 
LIST  IN  EEI/OIR  FILE. 


NT  ANALYSIS  MATRIX  RE 


pill 


ECTION  PLANNING  FILE 


3.1. 1.3  DETERMINE  SPECIFIC  INFORMATION 


1.  BREAK  EACH  EEI/OIR  INDICATOR 
INTO  SPECIFIC  INFORMATION  AND: 

a.  KEY  SPECIFIC  INFORMATION 
DERIVED  TO  THIS  APPROPRIATE 
INDICATOR. 

b.  ASSIGN  THE  PRIORITY  OF  THE 
ASSOCIATED  INDICATOR  TO  THE 
SPECIFIC  INFORMATION. 


2.  CHECK  FOR  AND  CROSS  REFERENCE 
DUPLICATE  ENTRIES  TO  ELIMINATE 
DUPLICATE  TASKING  AND  PROCESSING. 


3.  ADD  SPECIFIC  INFORMATION  TO  THE 
EEI/OIR  FILE.  THE  SPECIFIC 
INFORMATION  WILL  FORM  THE  BASIS 
FOR  SPECIFIC  ORDERS  AND  REQUESTS. 


3. 1.1.3  DETERMINE  SPECIFIC  INFORMATION 


1.  CALL  SCROLLABLE  INDICATOR 
LIST  ON  ONE  CRT,  AND 
SEARCHABLE/SCROLLABLE 
COLLECTION  PLANNING,  FILE 
ON  ADJACENT  CRT.  SCROLL 
THROUGH  EACH  INDICATOR. 

FOR  EACH  INDICATOR: 

a.  LOCATE  SPECIFIC  INFO 
ITEM  IN  COLLECTION 
PLANNING  FILE  "COPY 
OVER"  TO  APPEND  TO 
INDICATOR,  OR  KEY-IN 
SPECIFIC  INFO  DES¬ 
CRIPTION  TO  INDICATOR. 

b.  WHEN  FINISHED  WITH  ALL 
INDICATORS,  REVIEW  FOR 
DUPLICATES,  AND  RELEASE 
UPDATE/MAINTAIN  VER¬ 
SION  OF  INDICATOR  LIST 
TO  ACTIVE  FILE  STATUS. 

2.  SPECIFIC  INFORMATION  IS 
ADDED  AUTOMATICALLY  TO 
EEI/OIR  FILE. 


COLLECTION  PLANNING  FILE 
EEI/OIR  FILE! 


B-MIQSTURE  Interview  Guide 


This  interview  concerns  how  you  process  tactical  intelligence.  We're  going  to  show 
you  selected  task  step  descriptions  which  probably  are  familiar  to  you.  We’d  like  to 
ask  you  questions  about  these  steps,  to  help  us  develop  an  automated  system  that 
would  aid  intelligence  processing.  Pour  areas  of  aiding  are  considered:  (1)  cueing  and 
prompting  of  a  task  step,  (2)  recording  the  outcome  of  a  task  step,  (3)  handling 
intermediate  data  results  from  a  task  step,  and,  (4)  carrying  data  over  between  task 
steps. 

The  separate  pamphlet  that  you  have  displays  descriptions  of  tasks  performed  during 
tactical  intelligence  processing.  The  left  half  of  each  page  describes  steps  in 
current  versions  of  tasks  selected  and  adapted  from  the  CEWIOCAFAS  document.  The 
right  half  of  each  page  describes  steps  in  computer  dided  versions  of  the  same 
tasks. 

We  wish  to  get  your  opinions  about  several  aspects  of  these  descriptions  of  tasks. 
First  let’s  concentrate  on  current  versions  of  the  tasks;  but  we're  also  interested  in 
your  comments  about  the  automated  support  versions.  For  a  given  task  step,  only 
some  of  the  questions  in  this  Interview  guide  may  be  appropriate. 

Your  expert  opinion  as  an  intelligence  analyst  is  very  important  to  this  project.  The 
purpose  is  to  incorporate  your  experience  in  ideas  for  an  automated  system.  We’d 
like  to  remind  you  that  your  answers  to  our  questions  are  intended  only  for  use  in  this 
research. 
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Task  stap  being  considered  Is: _ 


Assume  the  performer  of  the  task  understands  the  task  step  described  and  its 
activities,  has  been  trained  on  them,  but  may  be  "rusty",  tired,  distracted,  or  badly 
overloaded.  For  this  task  step,  how  useful  is  the  description  AS  GIVEN  for  reminding 
the  performer  about  the  details  of  the  what,  why,  when,  and  where  of  activities  to 
be  performed  in  this  step? 

(1.)  How  useful  Is  this  task  step  description  for  cueing  and  prompting  the  performer 
about  what  to  do? 

1  2  3  4  5  6 


Extremely 

Useful 


Of  little 
use 


(2.)  Please  comment  on  the  LEVEL  OF  DESCRIPTION  of  the  task  step. 

1  2  3  4  5  6 

— -  - •-» - - -  — 

Too  General  Too  Detailed 


(3.)  Are  the  task  step  BOUNDARIES  (beginning,  end)  correct  in  relation  to  the  other 
task  steps  surrounding  It? 


1  2 

3  4 

5 

6 

Many  activities 

Task 

Many  activities 

in  this  step  should 

Boundaries 

in  other  steps 

be  put  in  other  steps. 

0.  K. 

should  be  put  in 
this  step. 

Comments: 
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/ 


Task  stap  being  considered  is _ 

(4.)  Can  the  activities  of  this  step  be  eliminated  if  neccessary? 

1  2  3  4  5  6 

Always  Sometimes  Never 

If  sometimes,  under  what  conditions? 


(5.)  Is  this  step  concerned  with  non-deferrable  actions,  (l.e.,  with  processing  crucial, 
perishable  data)? 


1 

2 

3  4 

5 

6 

Always 

Sometimes 

Never 

if  sometimes,  under  what  conditions? 


(6.)  Can  the  activities  of  this  step  be  shortened  or  degraded  if  neccessary? 


1 

2 

3  4 

5 

6 

Always 

Sometimes 

Never 

If  sometimes,  under  what  conditions? 


Comments: 
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Task  stap  being  considered  is: _ 

(7.)  How  demanding  is  this  task  step  in  requiring  your  time  and  thought? 

1  2  3  4  5  6 


Extremely  Extremely 

Demanding  Un-demanding 


(8.)  In  this  step,  the  number  of  different  items  needing  more  or  less  simultaneous 
attention  is: 

1  2  3  4  5  6 

Very  Large  Very  Small 


(9.)  In  this  step,  the  number  of  items  that  must  be  recalled  BV  MEMORY  is: 

1  2  3  4  5  6 

Very  Large  Very  Small 


Comments: 


Task  step  being  considered  is: _ 

(10.)  Is  this  step  dependent  on  the  completion  of  other  steps? 

Describe  prerequisite  step(s).  Why  is  it  prerequisite? 


In  the  above  question  we  asked  about  prerequisite  steps.  We  are  also  interested  in 
requirements  of  the  present  step  for  DATA  generated  in  earlier  steps.  With  respect 
to  the  task  step  under  consideration,  data  from  EARLIER  steps  in  this  task  may  or 
may  not  be  useful  or  neccessary  in  completing  the  present  step. 

(1 1.)  How  OFTEN  are  such  earlier-developed  data  NECCESSARY  to  the  present  step? 

1  2  3  4  5  6 


Always  Sometimes  Never 

If  appropriate,  describe  data  and  conditions: 


(12.)  How  EASY  is  It  to  identify  the  kinds  of  data  needed  from  earlier  steps? 

1  2  3  4  5  6 

Very  easy  v  Very  difficult 

If  appropriate,  discuss: 


(13.)  How  READILY  ACCESSIBLE  are  the  needed  data  from  earlier  steps  (e.g.,  in  sit- 


maps,  message  files,  your  memory,  etc.)? 

12  3  4 

5 

6 

Readily  accessible 

Inaccessible 

If  appropriate,  discuss: 
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Task  step  being  considered  Is: _ 

In  s  computer-automated  support  system  for  data  processing,  it  is  possible  to  pro¬ 
vide  means  for  "checking  off"  each  step  in  a  task  as  it  is  accomplished  and  to  keep 
a  record  of  the  outcome  of  that  step.  Such  a  capability  is  more  useful  for  some  task 
steps  than  for  others  and  useful  for  different  reasons. 

For  the  task  step  presently  under  consideration,  how  useful  would  such  capabilities 
be: 

(14.)  For  the  TASK  PERFORMER  DURING  PERFORMANCE  OF  OTHER  TASK  STEPS,  to  help 
in  remembering  the  step  as  accomplished,  deferred,  deleted,  short-cut,  etc.: 

1  2  3  4  5  0 


Highly  Useful  Of  little  use 


(16.)  For  the  TASK  PERFORMER,  for  LATER  review  and  critique  of  OWN  performance: 

1  2  3  4  5  6 

Highly  Useful  Of  little  use 


Comments: 


Task  step  being  considered  Is: _ 

For  the  task  step  under  consideration,  how  useful  would  It  be  to  have  the  following 
different  means  of  recording  the  outcomes  of  the  step? 

(16.)  In  order  to  store  a  future  reminder  to  you  of  outcomes  resulting  from  this  step, 
would  you  find  it  useful  to  have  a  displayed  list  of  possible  outcomes,  where  you 
could  check  off  ones  that  occur? 


1  2  3  4  6 

6 

Highly  useful 

Of  little  use 

(17.)  For  the  same  purpose  as  above,  how  useful  would 
(for  entering  numerical  values,  dates,  etc.)? 

a  fill-in-the-blank  format  be 

1  2  3  4  5 

6 

Highly  useful 

Of  little  use 

(18.)  For  commenting  about  outcomes  of  this  step,  how  useful  would  it  be  to  be  able 
to  store  your  own  typed  notes  for  future  reference? 

1  2  3  4  6 

6 

Highly  useful 

Of  little  use 

Comments: 
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C-Questionnaire  Rating  Distributions  by  Tasks 
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O-Transcription  of  Comments 


2. 1.2.1.  DETERMINE  SOURCE  RELIABILITY 
Q.  (1.)  Conments: 

Check  Black,  White,  Grey  List  for  personality  if  new  source. 


Q.  (2) 

No  mention  of  data  validity/verification  -  source  reliability  is  only 
half  -  other  half  relates  to  analysts  belief  that  the  information  is 
likely  to  be  correct  (other  indicators). 

q.  (2) 

Too  much  reading  of  an  operator  is  already  tried.  Just  one  key  word 
should  be  sufficient,  i.e.  reliability  - . 


Q.  (3.) 

Considering  that  at  least  a  minimum  of  training  has  been  received  this  step 
is  of  little  value  to  the  TAC  intelligence  officer. 


q.  (3.) 

I  have  not  seen  source  reliability  files  maintained. 


Q.  (3.) 

"Reliability"  is  a  very  subjective  judgement.  An  operator  would  not  normally  know 
the  reliability  of  a  particular  source.  He  would  have  to  be  told  by  the  expert  on 
that  particular  system  to  get  a  reliability  judgement  of  any  value  and  even  then  I 
am  not  certain  it  could  be  described  by  an  alpha-numeric. 


The  term  (or  criteria)  "important/not  important"  may  be  easier  to  use  than  "reliable" 
due  to  the  fact  that  no  judgement,  on  the  operator's  part  is  called  for.  An  operator 
must  only  record  into  the  machine  whether  or  not  his  boss  told  him  that  a  particular 
report  was  important  or  not. 


2. 1.2.1.  DETERMINE  SOURCE  RELIABILITY 

Q.  (3.) 

Should  be  an  integral  part  of  all  data  entry,  and  as  a  reminder,  tasks  should 
be  included  as  1st  step  in  the  others. 

Q.  (4.) 

Additional  information  may  negate  the  need  to  determine  reliability. 

Q.  (4.) 

If  sufficient  quantities  of  information  are  being  received  from  varied  sources, 
then  receipt  of  the  information  from  2  or  more  sources  will  serve  to  prove  its 
reliability. 

Q.  (4.) 

Usually  the  source  reliability  is  included  in  or  appended  to  incoming  message, 
unless  the  report  is  being  originated  by  the  using  unit. 

q.  (4.) 

#2  Time:  Do  decide,  do  assign  class  (eg.  A-l)  may  not  allow,  be  ample  - 
go  on  gut  feeling. 

Q.  (4.) 

There  are  certain  instances  in  which  it  does  not  matter  if  a  source  is 
extremely  reliable  or„not.  In  any  case  it  will  probably  be  recorded 
anyway. 

Q.  (4.) 

Checking  file  often  is  superfluous,  particularly  if  the  source  is  conwonly 
heard  from. 
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2. 1.2.1.  DETERMINE  SOURCE  RELIABILITY 

Q.  (4.) 

Reliability  of  source  is  very  important. 

Q.  (4) 

Only  if  source  is  unknown. 

Q.  (5.) 

If  using  unit  is  originating  a  perishable  report  and  needs  to  include  a 
standard  evaluation  grade  of  that  source  (i.e.  that  source  would  receive  the  same 
evaluation  from  any  agency  which  originated  the  report. 

Q.  (5.) 

Perishable  information  is  of  more  value  to  people  who  can  shoot  at  it  or 
interdict  that  target. 

Q.  (5.) 

Sources  are  flexible. 

(5. ) 

Source  is  most  important  on  spot-report  type  reports  which  are  almost  always  time 
critical . 


Q.  (6.)  Comments: 

Reliability  determination  is  an  important  step  in  evaluation  of  information.  Over 
a  period  of  time,  as  analysts  become  more  familiar  with  various  sources,  it  becomes 
more  and  more  of  a  mental  process.  Failure  to  properly  evaluate  source  reliability, 
or  take  it  into  account  can  lead  to  faulty  intelligence  production  and  loss  of 
creditability. 


D-3 


2, 1,2.1.  DETERMINE  SOURCE  RELIABILITY 


Q.  (6.) 

Same  as  four.  There  are  times  when  the  information  should  be  acted  upon,  at  least 
as  an  intelligence  indicator,  and  often  times  unnecessary  delay  may  result  in 
missing  a  key  piece  of  data  or  ooportunity. 


Q.  (6.) 

Only  if  step  one  yields  no  information  at  all. 

Q.  (6.) 

If  using  an  incoming  report  which  already  includes  this  data. 


Q.  (6.) 

Eliminate  step  2  if  not  enough  time. 

Q.  (7.) 

Depends  on  the  type  of  source  being  evaluated  i.e.  electronic  versus  humint. 

Q.  (9.) 

If  evaluating  source  informally  and  not  by  SOP  or  by  an  actual  computation  of  his 
verification  rate,  it  is  necessary  to  recall  entire  history  of  source,  his  origin, 
the  volume  and  level  of  his  information,  and  his  approximate  "batting  average"  in 
particular  areas  of  information,  (i.e.  a  source  may  provide  strong  and  reliable 
background  but  poor  current  intelligence  because  of  his  placement  or  access). 

Q.  (9.) 

Item  7  step  demanding  in  time  not  demanding  in  thought. 

0.  (9.) 

An  operator  would  not  normally  have  any  idea  of  what  is  going  on  in  an 
intelligence  operations  center.  He  should  not  have  to  remember  anything. 


2. 1.2.1. 


DETERMINE  SOURCE  RELIABILITY 


Q.  (9.)  Comments: 

The  operator,  if  given  sole  responsibility  would  have  to  have  experince. 

Reliability  of  Sources  is  a  flexible  thing.  It  is  also  based  on  near  real  time. 
Today  it  is  good,  tomorrow  it  is  not. 

The  data  base  would  have  to  be  re-set  daily  -  if  not  every  8  to  12  hours. 


Q.  (10.) 
No. 


0.  (11.) 

Depending  on  source  e.g.,  Humint  -  previous  reliable  reports  would  lend 
more  credance  to  the  source  as  reliable. 

Q.  (11.) 

As  above,  it  might  be  desirable  to  be  working  with  a  continuously  updated  source 
reliability.  (May  be  too  complex  or  misleading  at  tactical  level,  but  on  the  other 
hand,  might  inject  objectivity  into  system. 

Q.  (11.) 

To  build  a  reliability  base,  past  reports  must  have  been  evaluated. 


Q.  (11.) 

Comparison  of  data  from  sources  or  like  sources  is  beneficial  in  determining 
rel i ability. 

(11.) 

Dependent  on  the  "data-base"  cf  available  collectors  and  their  -eliability  in  the 
post,  but  this  is  generally  kept  in  the  heads  of  the  operators,  net  written. 
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2. 1.2.1. 


DETERMINE  SOURCE  RELIABILITY 


Q.  (11.) 

Each  item  of  data  submitted  by  a  source  can  be  used  to  increase  reliability  of 
source  designation  -  each  confirming  piece  of  information  raises  relative 
accuracy  of  the  information. 


Q.  (12.) 

Comparison  of  effects  on  source  (electron)  by  weather  for  example  would  estab¬ 
lish  a  trend. 

Q.  (12.) 

If  source  provided  intelligence  test  to  see  if  it  was  subsequently  proven, 
disproven  or  unknown. 


Q.  (12.) 

Data  is  often  unavailable  to  confirm  or  deny  a  report  or  prediction. 

Q.  (12.) 

Reliability  of  sources  is  not  a  stagnate  thing. 

Q.  (13.') 

It  is  a  judgement  on  the  part  of  the  analyst. 

(13.) 

Is  virtually  inaccessable  unless  the  operator  has  prior  experience  with  the 
collectors.  In  that  case,  he  must  rely  on  the  assessment  of  other  people. 

Q.  (13.) 

Dependent  upon  type  of  information  required. 
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2. 1.2.1. 


DETERMINE  SOURCE  RELIABILITY 


Q.  (13.) 

Certain  sources  will  be  closely  monitored  and  all  intelligence  from  them  recorded. 
The  difficulty  will  be  in  objectivity  determining  whether  intelligence  was  later: 
proven,  disproven,  or  indeterminate. 

Q.  (13.) 

If  date  is  very  old  it  becomes  1)  erased  off  a  situation  map,  2)  lost  in  a 
clerk's  journal  causing  much  wasted  time  in  trying  to  locate. 

Q.  (13.) 

Volume  of  data  under  consideration  and  number  of  sources  dealt  with  causes  informa¬ 
tion/data  to  get  lost  in  the  manual  shuffle  or  fuzzy  in  the  memory. 


(14.) 

Helpful  for  those  who  have  had  little  or  no  prior  experience. 

Q.  (14.) 

Of  little  use  except  as  a  data  bank  of  related  information. 

(15.) 

/ 

In  a  f1 el  a  environment,  the  "crif.que"  of  performance  is  winning  the  war.  A 
critique  is  normally  not  necessary  except  in  some  excercises. 

Q.  (15.)  Comments: 

Once  done,  it  is  too  late,  must  be  accomplished  as  we  go  along. 


Q.  (15.) 

i he  ability  to  recall  messages  for  source  and  reliability  could  areatlv  aid  the 
process  of  analysis  and  production  and  Building  intelligence  data  -  OB,  EOB, 

over  a'stion  or  loJgliS!'  "  SySte,"S  S,'g1nt>  phot1nt'  etc'  > 
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2. 1.2.1.  DETERMINE  SOURCE  RELIABILITY 


Q.  (18.) 

The  whole  process  of  evaluating  sources  and  accuracy  of  information  may  not 
be  as  applicable  in  this  scenario  as  might  be  imagined.  If  too  much  weight 
is  attached  to  these  codes,  the  information,  no  matter  how  valid,  may  be  overlooked 
in  analysis  should  it  be  assigned  a  lower  than  appropriate  rating.  Secondly, 
unlike  the  HUMINT  field  where  one  is  dealing  with  sources  which  may  have  cause 
to  distort  the  facts,  generally  the  Tactical  Intelligence  field  deals  witn  sources 
that  have  no  reason  for  misrepresentation.  Notable  exceptions  are  those  items 
derived  from  Refugees  and  PW's  and  the  like. 

0.  (18.)  Comments: 

Recall  of  notes  or  interpretation  is  necessary  but  not  often  used  because  of 
reams  of  paper  that  must  be  dug  through  -  memories  are  not  always  exact  and 
seldom  reliable  in  high  stress  situations. 

Q-  (18.) 

Useful  to  use  check-off  format.  Useful  to  somehow  store  outcome  to  update  “batting 
average"  as  mentioned  earlier. 


Q.  (18.) 

“Fill  in  the  blank"  type  data  is  really  of  little  use  in  assisting  in  the  deter¬ 
mination  of  reliability. 

P.  (18.)  Comments: 

As  a  step  in  preparation  for  verification  tasking  under  the  unit  collection 
p  i  an  • 
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2. 1.2.2.  POST  ELECTRONIC  ENEMY  ORDER  OF  BATTLE  MAP 


Q.  (3.)  Comments: 

I  would  change  the  sequence  to  key  on  the  most  important  areas  first: 

A.  LOC/TIME 

B.  Type  WPN  SYS. 

C .  Emi tter  type 

D.  Unit  ID 


Q.  (3.)  Comments: 

1.  At  division  level,  given  current  capabilities,  task  should  key  on  A.  Emitter 
type  (ELINT  only),  B.  Location  and  time,  activity  reported,  C.  weapons 
system  association,  D.  Unit  ID  or  level  of  CMD.  Note:  Location  function  is 
extreme! y  limited. 

2.  The  development  of  EW  order  of  battle  holdings  (data  base)  should  be 
interfaced/  incorporated  at  this  step. 


Q.  (7.) 

It  is  either  there  or  not. 


q.  0.) 


Basically,  you  are  concerned7 with  types,  capabilities,  associated  systems  and 
wnat  unit/echelon  is  the  system  normally  located. 

Q.  (11.) 

Collection  data  base,  EOB  holdings. 

Q.  (11.) 


Determines  trends  i.e.  the  movement  of  an  emitter  from  one  local 
may  signify  an  impending  movement. 


to  another 


Q.  (12.) 

Movement,  locations,  durations,  etc. 


2. 1.2. 2.  POST  ELECTRONIC  ENEMY  ORDER  OF  BATTLE  MAP 


q.  (is.) 

Useful  in  later  review  to  assist  in  deterging  trends. 

Q.  (16.) 

Cross  reference/ recall  of  specific  gaps  in  data  to  request  technical  support. 

q.  (17.) 

Filing  technical  data  related  to  report. 


q.  (18.) 

r!ports  resu1tin9  from  SIGINT  OPNS/  time  differentials/ 
collection  management  notes. 
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2. 1.2. 3.  POST  COLLATERAL  ENEMY  ORDER  OF  BATTLE 


Q.  (3.)  Comments: 

Rather  than  questions  as  posed  —  ask: 

Unit  size  _ 

Unit  type _ 

Unit  location _ 

Unit  designation _ 

Unit  activity^ _ _ 

Special  weapons? _ 

Time  located 


Q*  (3.)  Comnents: 

(2)  Might  add  "Source  Evaluation". 

Q.  (3.) 

/ 

A  time  designator  would  be  needed.  An  operator  should  not  be  a  button  pusher 
but  an  Intelligence  Analyzer. 

A  computer  setup  for  Intelligence  should  be  a  data  base  -  Time  frames. 
Indicators,  sources,  etc.  This  system  should  have  "recall",  correlation  made 
to  assimilate  past  intelligence  with  present-future  intelligence  to  assist  the 
intelligence  analyzer  in  his  work. 

Q.  (3.)  Conments: 

Steps  should  be  in  order  of  immediate  need/interest: 

Time  and  Event 
WPN  system 

Location  and  disposition 

Unit  ID 

Strength 
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2. 1.2. 3.  POST  COLLATERAL  ENEMY  ORDER  OF  BATTLE  (Cont.) 
Question  (3)  Cont. 


( 3 . )  Comnents : 

SLS-Steps  in  this  stop  are  out  of  sequence.  Should  be: 

d.  Time  and  Event  • 

e.  Weapon  system 

b.  Location  &  Disposition 

c.  Strength  &  Composition 

d.  Unit  ID 

Q.  (4.) 

Given  only  1  or  2  of  the  items  of  information  called  for,  information  would  still 
be  posted. 

Q.  (40 

Elements  of  this  procedure  may  have  to  be  omitted  for  iack  of  information, 
however  as  much  as  possible  must  be  completed. 

Q.  (4.) 

Step  E  -  Time  constraints  and  certainly  state  of  the  art  grease  pencil  and 
acetate  would  eliminate  this  from  cluttering  up  the  OB  map  -  type  of  unit  should 
key  knowleagaole  person  to  weapons  types. 


Q.  (5.) 

If  information  is  reliable  and  accurate  enough  for  use  as  targeting  data,  it  would 
be  considered  perishable.  Time  and  event  may  be  indicators  of  immiment  action,  • 
ergo:  perishable. 


(5.) 

Almost  always,  since  they  tend  to  be  "event  generated"  (e.g.  -  a  nuclear  strike). 
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2. 1.2. 3. 


POST  COLLATERAL  ENEMY  ORDER  OF  BATTLE 


for  this  step. 


Q.  (5.) 

At  division  Tevel  much  of  the  data  is  old  upon  receipt. 


Q.  (6.) 

Maps  are  generally  posted  with  unit  size,  type  and  DTG  of  observation  as  a 
minimum,  more  information  being  nice  to  have  but  omitted  for  lack  of  tine  or 
data. 

Q.  (6.)  Comments: 

This  step/information  is  the  meat  and  potatoes. 

The  amount  of  information  received  within  a  one  hour  period  in  a  Corps  Toe 
is  around  100-200  msgs.  (based  on  Reforger  78  figures).  One  operator  could 
not  keep  up  with  this  and  information  would  become  old  before  it  is  used. 


Q.  (6.) 

For  example  unit  Desig.  and  40  will  generally  give  me  the  basic  composition/ 
disposition.  The  number  and  type  of  events  will  guide  you  towards  approximate 
strengths  etc. 

Q.  (6.) 

Items  b,  d,  and  either  a  or  e  are  a  minimum.  If  OB  on  that  particular  unit  is 
already  established,  you  would  need  only  updated  b  and  d  and  battle  damage  to 
unit,  if  any.  Units  size  (a)  and  type  of  weapon  system  (e)  would  also  determine 
if  it  would  be  posted  on  a  division  SITMAP.  (i.e.  DTOC  would  not  normally  post 
a  pit  or  a  single  howitzer) 


Q.  (6.) 

Steps  C  &  E  *  time,  map  clutter,  stress. 
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2. 1.2. 3.  POST  COLLATERAL  ENEMY  ORDER  OF  BATTLE 


Q.  (6) 

See  4/5 

Q.  (6.) 

Only  if  information  is  of  a  catastrophic  nature  ("■flash''  -  initiation  of  hos¬ 
tilities,  use  of  C8R  weapons,  paradrop  in  friendly  rear  areas,  etc.). 


(8.)  Comments:  Ref.  g- 

Even  though  references  are  available,  the  knowledge  of  order  of  battle  must  be 
available  quickly,  since  intelligence  is  generally  short  fused.  This  is  an  area 
where  automation  could  simplify  and  shorten  the  time  needed  to  look  up  references. 

Q*  (9.)  comnents: 

Items  recalled  by  memory  include: 

a.  Previously  located  units,  to  detect  movement  may  be  more  in  "analysis" 
step  than  in  plotting  but  is  done  while  plotting  in  the  field. 

b.  Previously  identified^ units,  to  ID  partially  ID-ED  or  erroneously  ID-ED 
units  in  incoming  reports. 

c.  Template  of  en.  doctrine  to  assist  in  ID-ing  parcially/erroneously 
ID-ED  units  by  knowing  what  is  likely  to  be  there. 

Q.  (9.) 

?  Time/thought  requirement  increases  as  amount  of  information  on  map  increases. 
Biggest  difficulty  is  determining  whether  Bn  now  at  point  A  is  the  same  Bn 
previously  reported  at  point  B,  and  determining  if  that  unit  is  still  worth  postina 
(i.e.  what  have  they  lost?) 

Q.  (9.) 

The  above  questions  are  complicated  by  a  fast  moving,  central  European  scenario 
where  we  can  expect  actions  to  be  fast  and  furious. 
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2. 1.2. 3.  POST  COLLATERAL  ENEMY  ORDER  OF  BATTLE 


Q.  (9.)  Comments: 

Map  symbols  for  units  and  weapons,  OB  designations  after  a  time  become 
“second  nature"  to  analyst. 


q.  (io.) 

Apart  from  use  against  a  pre-existant  data  base,  no. 


Q.  (10.) 

This  step  is  complicated  with  a  combination  of  other  sources. 


Q.  do.) 

No  IFF  information  is  provided  without  requirement  for  local  collection  plannina 
and  tasking. 


q.  (u) 

Posting  is  in  my  opinion,  an  ini ti al  step  because  1)  If  intelligence  is  hot, 

•  1  Vo\Viral  t0  9raPhically  display  it  as  soon  as  possible  to  see  what  it  means 
and  2)  To  determine  what  further  processing  is  necessary  to  expand  on  it. 

q.  (11.) 

Never  necessary  almost  always  useful . 

q.  (ii.) 

All  source  analysis. 

q.  If  time  is  critical  and/or  redundant  sources  provide  supporting  data. 

| 
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2. 1.2-3. 


POST  COLLATERAL  ENEMY  ORDER  OF  BATTLE 


Q.  (12.) 

Previous  location  of  subject  unit  should  be  a  well  defined  term 


Q.  (12.) 

Again,  how  reliable. 


Q.  (13.) 

OB  generally  is  stored  in  a 
dependance  on  memory  (which 


file  (binder)  requiring  flipping  of  pages  or 
may  not  be  accurate  under  stress) 


0.  (13.) 


Strength  figures  are  often  harder  to  obtain,  but  are  also  least  necessary. 


Q.  (13.) 


If  memory  doesn't  key  you  to  some  particular  msg,  it  can  be  very  time  consuming 
to  find  data.  SITMAP  very  helpful,  but  gets  very  cluttered. 

Q.  (13.) 

Accessible  but  extremely  time  consuming. 


(13.) 

Order  of  battle  maps  are  posted  directly  from  the  incoming  reports. 

Q.  (14.) 

I  would  hope  that  in  this  particular  event  the  officer  or  OG  technician  will  not 
need  to  be  reminded. 

Q.  (14.) 

Data  can  be  annotated  as  posted/not  posted,  reported/not  reported  to  highter/lower 
etc. 
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2. 1.2. 3.  POST  COLLATERAL  ENEMY  ORDER  OF  BATTLE 


Q.  (15.)  Comments: 

Gen.  Of  the  data  required,  item  "a"  is  most  often  missing.  Should  have  some  way 
of  comparing  data  we  do  have  and  trying  to  match  with  existing  unit  in  OB  file. 
Task  as  stated  ("extract  &  "post")  is  simple.  Difficulty  comes  in  comparing  new 
data  with  old  in  "updating"  map,  rather  than  just  continually  posting  all  data. 

Q.  (15.)  Comments: 

Each  component  should  be  prompted  and  an  UNKNOWN  demanded  if  information 
unavailable  to  insure  completeness. 

On  display  only  unit  designation  (symbol)  and  location  need  to  be  routinely 
displayed,  other  data  can  stand-down  unless  called  up  (light  pen,  etc.) 

Q-  (12.)  Comments: 

This  is  a  very  basic  step  with  most  "analysis"  due  to  analyst  experience  and 
written  guidelines.  Once  credibility  is  assigned  a  piece  of  information  little 
need  exists  to  verify/recall  source  reliability.  Verification/recall  of  source 
for  confirmation  tasking/reporting  is  more  important. 


Q.  (15.)  Comments: 

/ 

RAM  for  history  of  units,  movements,  etc. 

Q.  (15.)  Comments: 

Once  reports  are  input,  subsequent  review  is  not  really  required. 

Q.  (15.)  Comnents: 

This  computer-auto-support  system  should  store  information/intelligence  to  allow 
ins <.ant  re-call. 

Q.  (15.)  Coronents: 

As  a  review  I  do  not  think  many  analysts  are  concerned  with  keeping  score  of  their 
correct  guesses.  However,  would  be  very  useful  in  recouping  for  stats  and  briefs. 
It  is  always  difficult  to  recall  exactly  what  occurred  six  hours  earlier  in  the 
battle. 
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2. 1.2, 3. 


POST  COLLATERAL  ENEMY  ORDER  OF  BATTLE 


q.  (18.)  Comments: 

Access  to  notes  and  messages  under  present  system  allow  reconstruction  of 
events  only  after  the  fact  and  with  much  deciphering  and  time,  e.g.  afte 
an  exercise. 

Q.  (18.)  . 

£«-v  OB  technician  hat  developed  Ms  own  particular  system  and  the  aM.H.y  •» 
Spot  my  notes,  consents  etc.  would  he  very  useful. 


q.  (is.) 

Useful  to  store  SJTMAP  as  a  whole  at  regular  intervals. 


Q.  (18.) 
Essential. 
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2. 1.2.4.  MAINTAIN  INTELLIGENCE  WORKBOOK 


Q.  (1.) 

Primarily  part  3  of  the  step  add  INTEL  SIT  MAP 
Q.  (3.)  Conments: 

Need  to  key  to  2. 1.2. 6.  so  automatic  check  of  doctrine  is  accomplished. 

Doctrinal  templates  should  also  have  key  indicator  list  -  i.e.  activate. 

Q.  (3.)  Comments: 

To  much  too  read  -  operator  would  not  have  time,  with  the  amount  of  information 
comoing  in  to  read. 

A  data  base  -  pre-set  of  Terrain  factors,  doctrinal  factors,  weather,  should  be 
set  up.  As  information  comes  in  an  analysis  could  refer  to  this  data  base,  analyze, 
post  and  react  to  information  being  received. 

Q.  (4.) 

Intelligence  workbooks  usually  get  reduced  to  spot  reports  files,  with  no  one 
with  time  to  work  on  the  workbooks. 

Q.  (5.) 

Once  EOB  map  is  posted,  the  workbook  is-  not  really  workable,  given  time 
constraints  on  personnel. 


Q.  (6.) 

WB  posted  when  time  available. 

Q.  (6.) 

Only  ’ f  step  one  yields  no  information  at  all. 

0.  (6.) 

Reduction  to  EOB  posting  on  map  is  almost  always  possible. 


2. 1,2.4.  MAINTAIN  INTELLIGENCE  WORKBOOK 
Q.  (7.) 

Depends  on  whether  or  not  information  is  raw  data  or  already  analyzed. 
Q-  (7.) 

Time,  not  thought. 

Q.  (8.) 

Report  dependent 

Q.  (9.) 

Report/situation  dependent. 


q.  (10.) 

No. 


Q.  Ul.) 

Indicators  come  in  one  at  a  time,  to  reach  a  conclusion  you  need  additional 
information. 

q.  (a.) 

'iust  be  able  to  recognize  patterns,  and  determine  when  data  becomes  obsolete. 

Q.  (12.) 

An  operator  would  need  to  be  an  intelligence  analysis, 

q.  (15.) 

It  may  be  too  late,  later, 
q.  (15.)  Comments: 

You  do  not  have  time  in  the  field  to  critique.  The  job  is  done  or  not  done. 
The  information  being  dealt  with  is  on  a  near-real  time  basis  and  is  thus  very 
peri shable. 
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2. 1.2.4.  MAINTAIN  INTELLIGENCE  WORKBOOK 
Q.  (18.)  Conroents: 

Intelligence  information  is  not  cut  and  dry.  It  is  dependent  on  weather,  terrain 
and  tactical -strategic  situations.  This  is  no  one  answer  to  any  given  set  of 
questions. 

Again  the  system  needed  by  the  intelligence  community  out  in  the  field  is  one  that 
is  a  data  base  -  so  that  re-call  and  information  assimilation  can  be  done. 

Q.  (18.)  Cont. 

A  system  which  is  based  on  immediate  input  of  time  sensitive  material  would 
be  useless. 

Q.  (18.) 

Let  analyst  record  his  ideas  so  it  will  key  him  if  other  data  comes. 
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2. 1.2. 5.  MAINTAIN  FRIENDLY  SITUATION  OVERLAY. 

Q.  (3.) 

I  think  that  this  Is  a  great  Idea  -  periodic  display  of  the  friendly  situation 
on  the  OB  map  would  be  a  tremendous  asset  however,  all  source  to  the  6-3  and  his 
office  would  create  a  cumbersome  situation  security  wise.  Data  which  Is  transmitted 
to  the  6  3  should  not  Include  compartmental  Intelligence. 


(3.)  Comments: 

There  will  be  a  security  problem  with  point  2  of  this  step.  Intelligence  Infor¬ 
mation  must  be  "sanitized"  before  posting  to  the  S3  map,  and  the  system  must 
be  secured  so  that  people  outside  the  EWIOC  cannot  gain  Information,  via  the 
computer,  which  contains  compartmented  Information. 

(4.) 

When  the  G2/G3  maps  are  situated  close  enough  so  that  both  can  be  seen  at  once. 
Q.  (4.) 

Focus  on  enemy  forces  Is  possible  with  friendly  boundaries  posted  for 
addresses  of  warnings  but  more  knowledge  of  friendly  dispositions  would 
make  warnings  timelier. 

Q.  (5.) 

Data  Is  old  at  division  except:  incorporation  of  plans  into  collection  efforts 
to  support  future  opinions. 

j 

(5.) 

Since  the  friendly/enemy  situation  can  change  rapidly,  and  decisions  are  made 
based  on  G2/G3  maps,  they  must  be  current. 


Q.  (5.) 

Boundaries  can  be  sufficient  at  times,  boundary  changes  being  conditions  where 
posting  Is  non-deferable. 
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2. 1.2.5.  MAINTAIN  FRIENDLY  SITUATION  OVERLAY 

(6.) 

Sometimes  -  Commanders  are  sometimes  briefed  from  the  knowledge  in  the  heads 
of  the  G2/G3  when  the  situation  changes  too  rapidly  to  post. 

Q.  (6.) 

Boundary  posting. 


Q.  (7.) 

This  is  a  manual  operation  largely  dependant  of  knowledge  of  friendly  tactics. 


Q.  (9.) 

We  can  expect  a  Central  European  conflict  to  be  extremely  fast  moving.  Movements 
and  locations  of  friendly  forces  must  be  known  to  all  staff  planners  on  a  con¬ 
stant  basis. 


Q.  (15.) 

Basically  for  the  same  reasons  as  task  2123. 


2. 1.2. 6.  MAINTAIN  TEMPLATING  FILES 
q.  (2.)  Conwents: 

Need  more  specific  actions  to  Input/ develop  templates. 
Q.  (3.)  Comments: 

The  operator  is  going  to  be  too  busy  to  read. 


Q.  (4.) 

Not  really  -  other  than  doctrinal  template  -  others  are  more  or  less 
progressive. 


Q.  (4.) 

For  good  analysis,  one  needs  a  base  to  work  from  IPB  is  that  base. 


Q.  (4.) 

In  current  type  systems. 


g.  (5.) 

Provide  cues  to  critical  times  and  places  on  battlefield  and  location  of 
possible  critical  codes. 


Q.  (5.) 

Intelligence  is  very  time  sensitive. 

Q.  (6.)  ~ _ ~ 

Actual  posting  of  templates  Is  very  time  consuming  -  using  event  or  decision 
theme  -  codes  are  usually  briefed  as  to  situation  and  postulated  situation 
which  would  require  action. 

Q.  (6.) 

Use  of  doctrinal  templates  to  make  situational  decisions/establish  decision 
points/criteria  in  lieu  of  Oecision  support  templates. 
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2. 1.2. 6  MAINTAIN  TEMPLATING  FILES 

Q.  (9.)  Comments: 

Every  bit  of  information  is  Important. 

Q.  (9.)  Comments: 

Many  things  are  based  on  memory  because  it  is  usually  too  time  consuming  to 
be  constantly  flipping  through  doctrinal  references  or  history  to  verify 
event  or  templates. 

Q.  (11.) 

In  building  OB  to  correlate  to  doctrine. 

Q.  (13.) 

The  operators  education  level,  i.e.  reading  ability,  comprehension  ability 
would  need  to  be  considered. 

Q*  (15.)  Comments: 

Instantaneous  display  of  templates  for  correlation  and  comparison  necessary. 


Q.  (15.) 

I  doubt  seriously  that  the  analyst  will  have  the  luxury  of  critiquing  his  own 
performance.  I  VAN  will  do  it  for  him. 


Q.  (15.) 

Time-factor  is  Important.  Given  the  amount  of  information,  etc.,  couid  the 
operator  get  through  all  the  tasks?  Probably  not.  Templating  should  be  in  a 
preplanned  data  base  file. 


Q.  (18.)  Conments: 

Templating  is  a  pre-planned  data  base  file.  Re-call  during  a  tactical  situation 
in  order  that  the  analysis  could  compare  items  is  important. 
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3. 1.1.2.  DETERMINE  INDICATORS 

Q.  (3.)  Comments: 

Step  3  vital,  most  priority. 

Q.  (4.) 

Poorly  developed  e  quation. 

Q.  (5.) 

In  fast  changing  current  situations. 


Q.  6.) 

As  In  5. 

Q.  (9.)  Comments: 

"As  you  sow,  so  shall  you  reap"  deserving  of  detailed  planning. 

Q.  (10.) 

Not  really. 

Q.  (11.) 

Possibly  can  develop  Indicators  by  ongoing  analysis  of  previous  enemy 
actions  to  detect  Idiosyncrasies. 

Q.  (12.) 

Hard  to  define  profitable  enemy  Idiosyncrasy. 
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3. 1.1.2.  DETERMINE  INDICATORS 


Q.  (18.) 

Might  be  able  to  use  In  writing  future  collections  plans. 

Conments: 

(16)  &  (18)  Best  to  display  a  matrix  menu,  allowing  collection  manager  to 
task  assets  in  fast  breaking  situation. 


3. 1.1.4.  REVIEW  THE  COLLECTION  PLANNING  FILE 


Q-  (4.) 

Step  C  *  depends  on  situation. 

Q.  (6.) 

Based  on  Procedure  -  intended  recipient  step  may  take  priority  with 
intermediary  really  short  time  later. 


Q.  (13.) 

Depends  on  the  Coll ecti on/Reporting 
automatically  get  re-transmi tted  or 
jog  system  for  input  (vice  versa). 


system  -  whether  or  not  messages 
the  lower  echelons'  must  constantly 
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3. 1.2.1.  DETERMINE  RESOURCE  AVAILABILITY 


Q.  (3.) 

1.  A  TOL  would  never  be  aware  of  the  status  (operational)  of  a  particular  system 
until  after  that  system  had  failed  to  gather  information  -  that  step  (#2)  would 
not  be  of  importance  nor  worth  the  time  to  keep  track  of  for  the  division  collection 
manager. 

2.  The  only  value  I  can  see  for  part  3  would  possibly  be  for  briefing  purposes. 

The  collection  manager  is  well  aware  of  what  assets  are  available  and  needs  no 
reminder  of  that  availability. 


(3.)  Comments: 

3.  3114  &  3121  are  reversed.  Cannot  do  collection  plapning  until  resource 

availability  is  determined. 


Q.  (4.) 

The  point  is  to  insure  that  one  collection  system  is  not  overloaded  or  relied 
on  too  much.  Need  to  consider  all  available  before  tasking. 


Q.  (5.) 

Need  to  confirm  information  from  one  source  in  a  timely  manner. 


(5.) 

Available  resources  change  constantly  &  this  list  must  be  current  .  (If  a 
Mphawk  has  been  shot  down,  it  can't  be  tasked  for  intelligence). 

Q.  (5.) 

If  one  resource  fails  and  task  must  be  given  quickly  to  another. 


D-29 


3. 1.2.1.  DETERMINE  RESOURCE  AVAILABILITY 


Q.  (6.) 

Seme  things  can  only  be  collected  or  confirmed  by  given  source.  Only  certain 
assets  are  available  in  time/area  required. 

Q.  (6.) 

Immediate  Intelligence  collection  task  may  not  allow  time  for  listing. 

Q.  (7.) 

Check  operating  status  may  take  time,  little  thought. 


Q.  (9.)  Conments: 

The  big  point  in  this  step  is  finding  out  what  is  available  to  do  the  job. 
Knowledge  of  system  capabilities  is  extremely  important  and  in  and  of  itself 
shorten  the  step. 


(9.)  Comments: 

Systems  availability  is  usually  kept  on  a  chart,  however,  since  situations  change 
rapidly,  this  may  not  always  be  up-to-date. 

y 

Q.  (11.) 

Need  current  data  of  what  equipment  is  destroyed,  damaged,  etc. 

Q.  (13.) 

Depends  on  how  the  data  base  is  kept  up  and  personal  knowledge  of  current 
situation. 

Q.  (13.) 

Difficult  to  get  all  resources  to  inform  DTOC  of  their  current  status. 
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3. 1.2.1.  DETERMINE  RESOURCE  AVAILABILITY 


Q.  U4.) 

To  prevent  over  loading  and/or  duplication  beyond  that  required. 

Q.  (14.) 

Should  be  able  to  note  when  equipment  is  already  being  used;  damaged; 
etc.  and  delete  when  destroyed. 

(18.)  Comments: 

If  a  systems  availability  flow  chart  could  be  automated,  it  could  be  corrected 
as  the  situation  changes  so  as  to  be  current. 
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3. 1.2.2.  DETERMINE  RESOURCE  CAPABILITY 


Q.  (3.)  Comments 

There  are  literally  stores  of  parameters  used  to  determine  which  source/system 
can  best  satisfy  a  particular  EEI/OIR,  etc.  Some  parameters  are: 

1.  Time  of  day  (lighting  conditions) 

2.  Tanget  weather  (rain,  fog,  etc.) 

3.  System  capabilities/limitations  (i.e.  given  a  particular  air  defense 
threat  can  an  aerial  platform  satisfy  your  requirements) 

4.  Criticality  of  the  information. 

5.  Etc. ,  etc. ,  etc. 

Q.  (4.) 

When  developing  original  collection  plan.  When  you  have  a  single  high  priority 
collection  task,  selection  of  source  may  be  done  without  listing. 

Q.  (5.) 

When  such  confirmation  of  other  source  or  suspected  information  is  needed. 

Q.  (6.) 

Same  situations  as  above,  but  only  if  step  has  been  previously  accomplished  en 
detail. 


Q.  (6.) 


If  there  is  for  instance  particularly  bad  weather  over  the  area  of  operations, 
then  all  aerial  platforms  may  be  hrounded  thus  eliminating  the  requirements  to 
consider  those  in  your  selection  process. 


Q.  (6.) 

On  the  spot  decisions  by  operations  officer. 

q.  do) 

Yes,  hundreds. 

Q.  (II.) 

Must  have  data  on  what  is  available  and  what  each  source  is  capable  of. 


i 
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3. 1.2.2.  DETERMINE  RESOURCE  CAPABILITY 


Q.  (12.) 

In  this  step  "available1*  rears  anything  which  can  be  used,  regardless  of  whether 
it  is  currently  engaged.  Next  step  would  be  assigning  resources,  in  which  case 
available  would  mean  free  to  be  used  at  this  particular  time.  "Available"in  the 
letter  sense  would  be  more  difficult  to  determine. 

Q.  (15.) 

Data  will  be  needed  when  resources  must  be  assigned  tasks.  If  possible  amount  of 
time  each  resource  is  tasked  can  be  tabulated  to  later  determine  how  successfully 
assets  were  managed. 

Q.  (17.) 

If,  for  instance,  you  wanted  to  withdraw  information  (for  briefing  purposes) 
of  the  capabilities  of  a  system  a  format  like  that  might  be  O.K.  For  operational 
purposes  of  no  use.  ~c - 


Q.  (16.  and  18.) 

Note  when  resource  is  tasked,  to  do  what,  when  completed,  and  results.  What  he 
did  and  the  results  could  be  cross  referenced  to  another  file,  not  detailed  in 
the  resource  cap/avail  listed.  Again,  amount  of  time,  #  of  tasks  could  be  noted. 


I 
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3. 1.2.3.  IDENTIFY  OTHER  SOURCES 
Q.  (2.) 

Too  large  a  source/collection  of  Information  to  be  made  compatible  with  IP. S.'s. 
Q.  (3.)  Comments: 

These  task  steps  should  be  listed  under  resource  availability. 


Q.  (6.) 

During  periods  of  intense  activity  the  procedure  to  reyiew  and  alleviate 
resources  to  gather  information  can  be  shortened  by  fast  on  the  spot 
decisions. 

Q.  (11.) 

If  reliability  factors  indicates  that  assets  are  not  functioning  properly,  more 
effort  can  be  placed  In  other  areas. 


Q.  (13.) 

The  only  reference  for  an  assets  reliability  to  my  knowledge  has  been  the  operator 
officer's  memory. 
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E-MIQSTURE  Experimenter's  Guide 

MQSTURE  EXPERIMENTER’S  GUIDE: 

The  tallowing  items  within  quotas  should  be  explained  to  each  participant.  Athough 
they  may  be  paraphrased  slightly  to  sound  less  formal,  all  of  the  information  should 
be  conveyed  to  each  participant  and  In  the  order  given  here.  Other,  ncn-quoted 
items,  are  tar  the  experimenter’s  benefit. 


GREETING  STEP:  Explain: 


a.  "You  are  one  of  a  large  group  of  participants.  The  exercise 
will  take  about  an  hour." 

b.  "This  is  not  a  test  of  your  abilities;  what  we  are  testing  is 
two  (Efferent  ways  to  enter  questions  into  a  computer  ter¬ 
minal.  Each  of  the  participants  enters  questions  using  both 
methods  of  entry.  Then  we  average  the  results  for  all  parti- 

•  dpents,  and  compare  the  averages  obtained  from  the  two 

methods." 

c.  "First  you  will  read  and  practice,  and  then  do  the  parts  of 
the  experiment." 

d.  "Don’t  worry  about  ’mistakes’,  this  Is  part  of  what  we  are 
testing  about  the  methods  of  entry.  We  won’t  ask  you  to 
do  the  test  tasks  until  your  practice  performance  looks  OK 
to  us." 

ORIENTATION  STEP: 

a.  "This  is  the  computer  terminal." 

b.  "It  communicates  to  you  through  the  screen.” 

c.  "You  communicate  to  it  through  the  keys." 

d.  Demonstrate  the  cursor  to  the  participant,  calling  It  by 
name. 

e.  In  a  setting  without  distractions,  give  participant  the  "PAR¬ 
TICIPANT  ORIENTATION  BOOKLET".  Indicate  that  the  booklet 
should  be  read  thoroughly,  and  the  four  practice  Items  at 
the  end  filled  In. 


PRACTICE  STEP: 


f.  When  participant  Is  finished  with  booklet,  answer  all  ques¬ 
tions  and  make  a  note  of  all  pertinent  questions. 

g.  Correct  and  discuss  the  two  practice  runs  with  participant. 

h.  Move  to  the  terminal: 

—  Log-on  using  Bell’s  ID;  [User:  osi040(]ER],  password: 
beffgffU  X  cd  testaTCm 

—  Empty  the  transactions  file:  [X  rm  transactions!  CR 1  ] 

—  Bring  up  MIQSTURE:  [X  mltfCTT,  bkfnfEXEC'UTEII 


a.  Seat  participant  at  terminal. 

b.  Show  participant  the  ian&I. 
buttons,  discuss  If  necessary. 


].  and  raA'CKSraCSI 
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c.  Ask  participant  to  sntsr  ths  first  practice  query  (Adams,C) 
In  ths  fBHn  form.  Help  with  oseh  stop  If  needed  (La.,qu 
fEOBCOTEl  to  got  1W-in  form). 

d.  Maks  surs  ths  participant  usss  propsr  uppsr  and  lowsr 
eass  characters,  and  also  that  thsra  la  no  spaca  batwsan 
ths  cossas  and  ths  first  initial.  Explain  that  ths  data 
ssareh  vakiss  must  bs  sntorsd  axaotty  as  shown  on  ths 
request  sups.  (TMs  is  not  discussed  in  tha  bookist  which 
ths  participant  has  just  road,  wa  will  got  bsttor  rsauits  if  it 
is  fuNy  explained  now). 

a.  Hava  participant  sntsr  second  practica  qusry  in  fill-in  form. 
Give  help  If  needed. 

f.  Ask  participant  to  sntsr  both  practica  queries  again,  but  in 
free- style  fons.  Again,  coach:  (La^  qu  is  fallowed  by  ona 
or  aero  Monks  and  than  ths  free -style  key-in  of  ths 
query).  Also,  nets  that  ths  "and"  must  have  a  space 
before  and  after  it  whan  typed  in  fre e-style. 

g.  Explain  that  the  participant  will  be  given  slips  of  paper  with 
questions  on  each  one  which  will  be  submitted  to  the  com¬ 
puter.  Each  slip  of  paper  has  the  query  in  plain  English,  and 
a  apace  for  the  answer. 

CONTROL  PERFORMANCE  STEP: 

a.  Give  tha  participant  Question  slip  #1,  simultaneously  hitting 
"READ*  on  ths  terminal. 

b.  Maks  surs  that  tha  participant  is  entering  the  query  in  the 
fiHn  form. 

e.  Repeat  for  sfipe  2  thru  6. 

VARIABLE  TREATMENTS  PERFORMANCE  STEP: 

a.  Participants  are  assigned  to  treatments  1,  2,  or  3  in  the 
sequence  in  which  they  appear  for  the  experiment  (i.e., 
1,2,3,1,2,3,etc.).  The  MIQSTURE  simulation  system  is  set 
by  tha  experimenter  to  one  of  the  three  treatment  condi¬ 
tions  for  the  variables  treatments  performance  step  as  fol¬ 
lows: 


—  Treatment  1:  no  change  from  previous  steps. 

—  Treatment  2s  bkfe  IEXECUIEI  (history  window  filled  ele¬ 
ment  at  a  tiam). 


—  Treatment  3:  bkfl 
at  a  time). 


L-LL4sIllJ 


(history  window  filled  line 


b.  Tefl  the  participant  that  another  5  questions  will  now  be 
given,  and  that  each  one  should  be  read  and  the  query 
entered  in  the  flIHn  format.  For  treatments  two  and  three, 
make  the  following  additional  explanations: 

—  Treatment  2:  "I  have  turned  on  the  transactions  window 
and  the  transactions  history  windows  (point).  Ac  you 
type  the  query  into  the  fRMn  form,  It  will  now  be 
automatically  translated  Into  the  free-styte  key-in  form 
and  wflt  appear  in  the  transactions  window,  a  term  at  a 
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thae.  Whan  you  ILHfCU  II)  tha  query,  tha  fraa-ctyta 
key-in  form  of  tha  full  quary  wffl  also  appaar  In  tha  Wa¬ 
tery  window.  In  this  way  you  can  compare  tha  fUl-in  and 
tha  free  styla  forms  for  antartng  a  quary.  This  is  helpful 
as  a  reminder,  because  later  you  are  going  to  enter 
queries  by  using  the  free-styie  form." 

—  Treatment  3:  "I  have  turned  on  the  query  history  win¬ 
dow  (point].  As  you  type  the  query  Into  the  fMl-ln  form. 
It  will  be  automatical^  translated  Into  the  free-styie 
key-in  form.  When  you  IEXECUTI1  tha  quary,  the  free¬ 
style  ksy-in  form  of  the  full  query  win  also  appear  in  the 
history  window.  In  this  way  you  can  compare  the  m-in 
and  the  free  style  forms  for  entering  a  query.  This  is 
helpful  as  a  reminder,  because  later  you  are  going  to 
enter  queries  by  using  tha  free-styie  form.11 

e.  fitve  participant  aNp  0  and  simultaneously  hit  READ,  making 
sure  participant  la  entering  the  query  in  the  fUl-in  form. 

d.  When  participant  has  EXECUTED  and  written  down  answer 
an  sflp,  say:  "Notice  the  free-styie  key-in  form  of  the  query 
you  just  entered  as  it  is  shown  in  the  history  window." 

CRITERION  PERFORMANCE  STEP: 

a.  Turn  off  history  display:  fbfcfnlPgCUTgn 

b.  Instruct  the  participant: 

—  "New  I'd  Ike  you  to  enter  some  qumrtms  by  using  the 
free  style  key-in  format  in  the  same  manner  you  did 
ssrlnr  during  the  origins!  practice  session.” 

e.  Cycle  slips  1 1  thru  1 6  In  the  same  manner  as  described  for 
previous  steps. 

d.  Make  sure  that  the  participant  Is  entering  the  queries  In 
free-styie  form. 

e.  After  finishing  the  fifteenth  query,  exit  MIQSTURE  (sx  - 
[EXECUTED-  Collect  the  1 5  query  request  slips  and  staple, 
writs  participant's  name  on  first  slip.  Also  writ* 
participant's  phone  number  or  moil  address  on  the  first  slip 
If  the  participant  wishes  to  hear  later  about  the  results  of 
the  experiaient.  Place  clips  In  MIQ  PARTICIPANT  RESULTS 
foidar.  Thank  and  dismiss  participant. 

MAINTENANCE  CYCLE: 

a.  Print  the  transactions  file  [X  print  transactions  rCTTl  writs 
participant's  name  an  tits  transactions  printout,  and  place 
In  MIQ  PARTICIPANT  RESULTS  folder. 

b.  After  assuring  that  the  transaction  fils  printout  is  complete, 
enter  two  commands: 

1.  [%  cat  transactions  »  srchlvsfCHIl 

2.  [X  rm  transsctionsfCffn 
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F-Partlcipant  Orientation  Booklet 


I 

|  PARTICIPANT  ORIENTATION  BOOKLET 

: 

The  exercise  you  are  about  to  help  us  with  is  an  experiment  on  ways  to  use  a  con* 
puter  terminal  to  ask  questions  of  a  file  of  data  stored  in  the  computer.  Two  ways  of 
submitting  questions  to  the  computer  system  are  being  compared.  In  the  experiment, 
you  wilt  convert  written  questions  givan  to  you  into  queries  that  the  machine  under¬ 
stands  and  submit  them  to  the  computer  through  the  terminal.  The  computer  will 
respond  with  the  number  of  records  It  finds  that  correspond  to  each  query. 


Here's  an  example.  Suppose  that  a  computer  file  of  information  about  employees 
working  for  a  company  is  maintained  for  various  uses  within  the  company.  Here  is 
what  the  information  on  one  parson  in  the  file  might  look  like: 


name 
(name) 
Jones,  B 

address 

(addr) 

12345  Main _ 

city 

(city) 

Encino _ 

state 

(atat) 

c* 

piii 

dept 

(dept) 

099 

1 2 -AN 

16- AN 

10-A 

< 

> 

04 

5-N 

3-N  ; 

This  is  one  record  aoout  one  person.  The  individual  divisions  of  information 
(separated  by  vertical  linerf)  in  each  employee’s  record  are  called  the  date  elements. 
Note  that  the  following  Items  are  shown  for  each  data  element;  from  top  to  bottom: 

a.  the  full  data  element  name  (name,  address,  city,  etc.) 

b.  name  abbreviation  (the  four  characters  in  parentheses) 

c.  apace  for  the  element  value  (uaing  underscores) 

d.  allowable  format  (1 2 -AN  means  twelve  alphabetic  or  numeric  characters  at  most 
may  be  typed  in;  10-A,  ten  alphabetic  charactera  only;  3-N,  three  numeric  only) 

The  information  in  a  data  element  is  called  the  data  element  value.  For  example, 
Jones,  B  is  a  possible  value  for  the  (name)  data  element,  Encino  Is  a  value  for  the 
(etty)  data  element. 


Remember: 


1 .  A  data  record  is  s  group  of  aata  alemsnts  about  on e  particular  thing. 

2.  A  data  adamant  is  a  singla  typa  of  information  in  a  data  racord. 


In  this  axarciaa,  you  will  ba  querying  (searching)  a  fite  of  parsonnal  records  like  the 
example.  Thera  are  two  ways  to  query  the  file  of  records  in  the  experimental  sys¬ 
tem  you  will  be  using:  1 .  free  key-in  style  and  2.  fill-in  the  form. 

Here  is  how  a  user  would  enter  a  query  using  the  free  key-in  style.  Suppose  you 
wish  to  find  the  number  of  records  in  the  file  for  people  who  both:  1.  live  in  Er.olr.o, 
ana  2.  have  4  years  on  the  joo.  You  wouid: 

1.  type  In  the  following  string  of  characters  and  spaces: 

qu  (clty)Encino  and  (yotj)4 

2.  push  the  button  marked  [EXECUTC]. 

e  The  "qu"  tells  the  computer  system  that  a  query  follows  on  that  line. 

e  "(city)Encino"  tells  the 'computer  to  count  the  number  of  records  in  the  file  for 
employees  who  live  In  Encino.  (Notice  that  there  is  no  space  between  the  ele¬ 
ment  abbreviation  and  the  data  value  following  It.) 

e  "(yotJ)4"  tells  the  computer  to  count  the  records  for  employees  with  four  years 
on  the  Job. 

e  Finally,  the  "and"  which  links  the  request  of  the  two  different  elements  (city, 
years  on  the  Job)  tells  the  computer  "count  ONLY  THOSE  records  In  which  ’(c‘ty)’ 
is  Encino  end  '(years  on  the  job)’  is  four." 

e  (The  message  returned  by  the  computer  Is:  "2  HITS"  for  the  experimental  cats 
file.) 
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The  other  way  to  enter  a  query  Is  through  the  fill-in  form.  The  fill-in  form  iocks  just 


like  the  sample  record  shown  earlier  In  the  booklet,  but  without  data  velues  in  the 
element  boxes.  The  figure  below  provides  an  example  of  the  fill-in  form  display,  'he 
fill-in  form  is  requested  from  the  machine  by  typing  "qu"  and  pushing  the 
In  response  to  the  "qu  lEXtecuffel"  command  the  fill-in  form  appears,  and  the  cursor 
moves  to  the  first  data  position  of  the  first  element.  (The  cursor  is  a  small  bright 
rectangle  on  the  screen  that  shows  where  the  next  letter  or  space  will  appear  when 
typed  by  you.  The  cursor  can  be  moved  by  the  alphabetic  and  numeric  keys,  the 
space  bar,  and  backspace  in  the  free  key-in  style  and  may  also  be  moved  by  the 
special  "and"  and  "or"  keys  in  the  fill-in  the  form  style.)  - 


In  the  example,  the  query  is  the  same  one  used  earlier,  (that  is,  employees  living  in 
Eneino  with  four  years  on  the  job,  without  regard  for  what  might  be  in  the  other  ele¬ 
ments  like  name  or  address  ). 


name 

address 

city 

state 

2tP 

dept 

years 

(name) 

(addr) 

(city) 

Eneino _ 

(stat) 

(ripe) 

(dept) 

(yotj) 

4. 

12-AN 

1 6-AN 

10-A 

2-A 

S-N 

3-N 

2-M 

In  this  form,  the  user  should  fill-in  only  those  boxes  needed  for  query.  The  AND  key 
is  depressed  to  move  the  cursor  to  the  right  from  one  element  box  to  the  next.  If 
the  records  for  all  of  the  people  who  live  in  Eneino  are  needed,  move  the  cursor  to 
the  element  box  labeled  "city"  and  type  in  Eneino.  Additionally,  If  only  the  persons 
from  Eneino  with  four  years  on  the  job  are  desired  to  be  Included  In  the  count,  the 
user  should  push  the  "and"  button  the  number  of  times  It  takes  to  move  the  cursor 
forward  to  the  first  space  In  the  "years"  element  box  and  type  In  “4".  Then  to  start 
the  operation  of  searching  by  the  computer,  the  user  must  give  the  query  to  the 
computer  by  depressing  the  lEXfedUTc]  button. 
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Again,  the  massage  returned  by  the  computer  is  "2  HITS". 


You  will  be  esked  to  da  both  types  of  query  entry.  That  Is,  you  will  be  entering  seme 

queries  via  freestyle  key-in  and  others  through  a  fiH-in-ihe-blsnk  form. 

The  sequence: 

e  The  interviewer  will  make  sure  the  computer  terminal  is  ready  to  be  used. 

e  As  you  are  given  each  question  on  a  piece  of  paper,  immediately  depress  the  but¬ 
ton  marked  "READ". 

e  Study  the  question,  and  when  you  feel  ycu  know  how  to  enter  a  query,  proceed. 
If  you  type  a  character  that  you  didn’t  mean  to  type,  the  "backspace"  key  is 
used  to  go  back  to  the  point  from  which  you  want  to  continue.  After  you  have 
entered  the  query  to  your  satisfaction,  push  [EXECUTE]. 

e  Record  the  number  of  “HITS"  and  return  the  piece  of  paper. 
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Now  for  some  practice.  Please  compose  the  four  queries  asked  for  below.  (When 
you  have  finished  all  four,  you  may  look  on  the  last  page  for  the  correct  answers). 
Using  the  fill-in  form,  find  out  If  there  are  any  employees  in  the  file  with  the  name  of 
■Adams.C". 


name 

address 

city 

state 

zip 

dept 

years 

(name) 

(addr) 

(city) 

(stat) 

(zipc) 

(dept) 

(yotj) 

12-AN 

16- AN 

10-A 

2-A 

5-N 

3-N 

2-N 

Compose  a  query  in  a  f Ill-in  form,  to  find  the  number  of  employees  in  department 
097  with  4  years  on  the  job. 


name 

address 

city 

state 

zip 

dept 

years 

(name) 

(addr) 

(city) 

(stat) 

(zipc) 

(dept) 

(yotj) 

12-AN 

16- AN 

10-A 

2-A 

5-N 

3-N 

2-N 

Enter  the  two  proceeding  questions  in  freestyle  key-in  form.  Print  the  query  as  you 
think  it  should  be  typed  in  the  foliowing  box. 

The  number  of  employees  in  the  file  named  "Adams.C", 


The  number  of  employees  in  department  097  with  4  years  on  the  job, 


r 


Using  the  fill-in  form,  find  out  If  there  are  any  employees  in  the  file  with  the  name  of 
"Adams.C". 


name 

address 

city 

state 

zip 

(name) 
Adaas,  C  _ 

(addr) 

(city) 

(stat) 

(zipc) 

BS«hS 

12-AN 

15-AN 

10-A 

2-A 

5-N 

I  EXECUTE! 


Compose  a  query  in  a  fill-in  form,  to  find  the  number  of  employees  in  department 
007  with  4  years  on  the  job. 


name 

address 

city 

state  I 

zip  | 

dspt  ; 

(name) 

(addr) 

(city) 

(stat) 

(zipc) 

(dsot)  1 
097  | 

12-AN 

15-AN 

10-A 

2-A 

5-N 

3-N  ] 

I  EXECUTE! 


Enter  the  two  proceeding  questions  In  freestyle  key-in  form.  Print  the  query  as  ycu 
think  It  should  be  typed  in  the  following  box. 

The  number  of  employees  in  the  file  named  "Adams.C", 
qu  (name)Adams.C  [EXECUTE] 

The  number  of  employees  in  department  097  with  4  years  on  the  job, 

I  qu  (dept)097  and  (yotj)4  [EXECUTE] 
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G-Test  Questions  Set 


1. 


2. 


3. 


4. 


5. 


6. 


7. 


8. 


9. 


10. 


11. 


12. 


13. 


14. 


16. 


Enter  a  query  in  the  fill-in  form  for  the  fc'lowing: 

The  number  of  persons  recorded  in  the  file  who  live  in  Oxnard  is _ . 

Enter  a  query  in  the  fill-in  form  for  the  following: 

How  many  persons  named  ''Ames,?"  are  in  the  file?  Answer _ 

Enter  a  query  in  the  fill-in  form  for  the  following: 

How  many  residents  of  Zip  code  91222  have  10  years  on  the  job?  An swer _ 

Enter  a  query  in  the  fill-in  form  for  the  following: 

Does  anyone  in  department  098  live  in  Burbank?  Circle  one:  yes  no 
Enter  a  query  in  the  fill-in  form  for  the  following: 

How  many  California  (Co)  residents  with  3  yeers  experience  ere  in  "'e? 

Answer _ 

Enter  a  query  in  the  fill-in  form  for  the  following: 

Does  the  city  of  Smog  View  have  a  Zip  code  of  91334?  Circle  one:  yes  no 
Enter  a  query  in  the  fill-in  form  for  the  following: 

How  many  residents  of  the  city  of  Simi  are  in  department  098?  Answer _ 

Enter  a  query  in  the  fill-in  form  for  the  following: 

Count  the  number  of  employees  living  in  Topanga  and  on  the  job  10  years. 
Answer _ 


Enter  a  query  in  the  fill-in  form  for  the  following: 

Does  anyone  by  the  name  of  11  Bush, L"  live  at  31  Thornyrose?  Circle  one:  yes  no 
Enter  a  query  in  the  fill-in  form  for  the  following: 

How  many  residents  of  Zip  code  91364  have  3  years  experience?  Answer, _ 

Enter  a  query  in  the  freestyle  form  for  the  following: 

How  many  Arizona  residents  (Az)  of  department  091  are  there?  Answer _ 

Enter  a  query  in  the  freestyle  form  for  the  following: 

Does  a  person  by  the  name  of  "Marr,A"  work  in  department  096?  Answer _ 

Enter  a  query  in  the  freestyle  form  for  the  following: 

Does  Burbank  have  a  Zip  code  of  91235?  Circle  one:  yes  no 
Enter  a  query  in  the  freestyle  form  for  the  following: 

Is  there  anyone  with  4  years  on  the  job  from  Burbank?  Circle  one:  yes  no 
Enter  a  query  in  the  freestyle  form  for  the  following: 

How  many  persons  named  "Blrd.R",  live  in  the  city  of  Woodland  and  have  9  years 
on  the  job?  Answer _ 
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H-MIQSTUF.E  Screen  Image 


The  accompanying  photograph  of  the  MIQSTURE  screen  shows 
the  display: 


the  various  windows  in 


1 .  The  top  window,  starting  TR8-.qu,  is  the  transaction  history  window.  In  the  photo¬ 
graph  it  contains  a  record  of  the  previous  query. 

2.  The  next  line  contains  three  status  windows.  The  file  status  window  indicates 
that  the  file-in-force  is  the  "people"  file.  The  transaction  status  window  in  the 
middle  indicates  that  there  Is  a  one  record  "hit"  for  the  previous  query  displayed 
in  the  transaction  history  window. 


3.  The  third  line  contains  the  current  transaction  window,  and  indicates  that  the 
current  transaction  (No  9)  is  to  be  a  query. 

4.  The  fourth  window  is  the  "data"  window,  and  contains  in  this  instance  the  fill-in 
format  for  the  record  type  in  the  "people"  file.  Note  that  the  user  is  in  the  pro¬ 
cess  of  entering  the  query  for  the  question  item  "Does  Burbank  have  a  Zip  code 
of  91235?”  (the  "35"  has  not  been  entered  yet). 
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